GitHub topics

llm-inference

50% of these repos are in LLM Infrastructure, 10% in Artificial Intelligence.

Repositories

20

Total stars

35,592

Trending

ruvnet

RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB built in Rust.

Created Nov 2025

ai ai-ocr attention-mechanism gnn gnn-model
Total4,713
30 days +112
spiceai

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

Created Aug 2021

artificial-intelligence data data-federation developers full-text-search
Total3,160
30 days +62
katanemo

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

Created Jul 2024

ai-gateway ai-gateway-support envoy envoyproxy gateway
Total7,118
30 days +29

Top

katanemo

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

Created Jul 2024

ai-gateway ai-gateway-support envoy envoyproxy gateway
Total7,118
ruvnet

RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB built in Rust.

Created Nov 2025

ai ai-ocr attention-mechanism gnn gnn-model
Total4,713
spiceai

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

Created Aug 2021

artificial-intelligence data data-federation developers full-text-search
Total3,160
Epistates

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Created Jan 2026

ai ane apple-silicon deep-learning distillation
Total333
OrbitInfra

OrbitKV is a High Performance KV Cache System for LLM Inference.

Created Mar 2025

attention kv-cache kv-cache-offloading llm-inference llm-serving
Total284
gpillon

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

Created Sep 2026

ada ampere blackwell coding-agents cuda
Total67

Related

All topics →