GitHub topics

vllm

76% of these repos are in LLM Infrastructure.

Repositories

13

Total stars

12,021

Trending

smg-project

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Created Nov 2025

anthropic anthropic-api chat claude gemini
Total732
30 days +85
OrbitInfra

OrbitKV is a High Performance KV Cache System for LLM Inference.

Created Mar 2025

attention kv-cache kv-cache-offloading llm-inference llm-serving
Total284
30 days +10
novitalabs

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Created Jan 2026

inference kv-cache llm vllm
Total221
30 days +3

Top

smg-project

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Created Nov 2025

anthropic anthropic-api chat claude gemini
Total732
OrbitInfra

OrbitKV is a High Performance KV Cache System for LLM Inference.

Created Mar 2025

attention kv-cache kv-cache-offloading llm-inference llm-serving
Total284
novitalabs

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Created Jan 2026

inference kv-cache llm vllm
Total221
azrtydxb

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.

Created Aug 2026

Total109
Netis

Agent and LLM API performance monitoring via network packet probe. Measures performance of OpenClaw, Claude, Codex, DeepAgents and more — deployed on the provider side, no SDK changes required.

Created Apr 2026

agentic-ai ai-agent-development ai-observability libpcap litellm
Total104
Dyluhn

Shape-specialized ROCm inference for Qwen3.8 Flash Next on RDNA4

Created Aug 2026

gguf qwen rdna4 rocm speculative-decoding
Total69
soapbucket

Open source Enterprise AI Gateway for API, MCP and agent, and AI model traffic. One Apache-2.0 binary: 72 native providers behind an OpenAI-compatible API, or serve vLLM and llama.cpp on your own GPUs. Keys, budgets, guardrails, semantic cache, WAF.

Created Apr 2026

Total54

Related

All topics →