Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,596

Last 30 days

+5,823

smg-project

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Created Nov 2025

anthropic anthropic-api chat claude gemini
Total735
30 days +76
OrbitInfra

OrbitKV is a High Performance KV Cache System for LLM Inference.

Created Mar 2025

attention kv-cache kv-cache-offloading llm-inference llm-serving
Total284
30 days +10
novitalabs

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Created Jan 2026

inference kv-cache llm vllm
Total221
30 days +3
azrtydxb

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.

Created Aug 2026

Total109
30 days +3
Dyluhn

Shape-specialized ROCm inference for Qwen3.8 Flash Next on RDNA4

Created Aug 2026

gguf qwen rdna4 rocm speculative-decoding
Total69
30 days +3
soapbucket

Open source Enterprise AI Gateway for API, MCP and agent, and AI model traffic. One Apache-2.0 binary: 72 native providers behind an OpenAI-compatible API, or serve vLLM and llama.cpp on your own GPUs. Keys, budgets, guardrails, semantic cache, WAF.

Created Apr 2026

Total54
30 days +0