Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,559

Last 30 days

+5,730

winfunc

A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.

Created Jan 2025

ai anthropic anthropic-claude api chain-of-thought
Total5,352
30 days +9
microsoft

AICI: Prompts as (Wasm) Programs

Created Sep 2023

ai inference language-model llm llm-framework
Total2,072
30 days +2
intentee

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Created Apr 2024

Total1,694
30 days +14
chrishayuk

Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.

Created Mar 2026

huggingface inference llm mechanistic-interpretability metal
Total1,194
30 days +25
smg-project

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Created Nov 2025

anthropic anthropic-api chat claude gemini
Total734
30 days +76
Ai00-X

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.

Created Jul 2023

ai aiagents chatgpt chatgpt-api embed
Total624
30 days +2
automataIA

GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration

Created Sep 2025

Total540
30 days +7
eliasbiondo

🕷️ The fastest open-source AI Gateway and LLM Gateway — one binary with built-in console to route, secure, budget and observe every LLM call across OpenAI, Anthropic, Gemini, Bedrock and more.

Created Oct 2026

a2a ai-gateway anthropic api-gateway bifrost-alternative
Total429
30 days +365
llamastash

Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.

Created May 2026

ai gguf llamacpp llm lmstudio
Total304
30 days +77
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total298
30 days +291
arcships

Unified LLM access layer in Rust — one API for 325 AI providers. 统一 LLM 服务接入层:一套 API 接入 325 家 AI 服务商。

Created Jul 2026

Total243
30 days +41
giannisanni

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Created Jul 2026

Total239
30 days +9
GetBusbar

The execution control plane for AI agents. Govern every model request, MCP tool call, A2A delegation, and downstream action—while keeping your native protocols intact: six native LLM protocols in, six native LLM protocols out.

Created May 2026

ai-gateway anthropic bedrock circuit-breaker claude
Total226
30 days +30
novitalabs

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Created Jan 2026

inference kv-cache llm vllm
Total221
30 days +3
matrixarkai

Open-source Rust TemporalStore for LLM context management, temporal features, and risk control

Created Jun 2026

ai-infrastructure context-engineering context-runtime feature-store llm
Total212
30 days +0
ferrumox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Created Mar 2026

Total191
30 days +2
AprilNEA

Turn AI subscriptions into standard API endpoints. OpenAI & Anthropic compatible gateway — for Amp Code, Cursor, Factory CLI, and more.

Created Feb 2026

ai-gateway amp anthropic-api claude copilot
Total141
30 days +3
RustyRAG

Production-grade RAG API built in Rust. Hybrid search with HNSW dense vectors and BM25 sparse matching, cross-encoder reranking, layout-aware document extraction via Docling, and 94.5% accuracy on Open RAG Bench. Powered by Cerebras, Groq, Milvus, and Jina AI.

Created Feb 2026

actix-web cerebras groq jinaai llm
Total139
30 days +1
lsuen

SUGT - local AI gateway & skills console. Centralize LLM API keys, one OpenAI/Anthropic-compatible proxy for Claude Code, Codex & OpenCode. Homebrew & winget. Fully open source (Apache-2.0).

Created Jul 2026

Total138
30 days +128
noumena-labs

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.

Created Mar 2026

Total125
30 days +3
AlephantAI

Alephant is an open-source AI Agent Gateway for routing, tracking, and controlling LLM usage across AI agents, members, and workflows, and for publishing agent capabilities as paid endpoints with x402 and MPP payment rails.

Created May 2026

agent ai ai-agent ai-gateways claude
Total119
30 days +4
azrtydxb

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.

Created Aug 2026

Total109
30 days +3
enapt

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

ai anthropic-api candle chatgpt-alternative claude-code
Total90
30 days +30
midagedev

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

Created Sep 2026

Total78
30 days +31
gpillon

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

Created Sep 2026

ada ampere blackwell coding-agents cuda
Total68
30 days +0
I3K-IT

🚀 100% local RAG system with one-command setup. Your data never leaves your server.

Created Nov 2025

binary docker document-ai enterprise eullm
Total59
30 days +0
soapbucket

Open source Enterprise AI Gateway for API, MCP and agent, and AI model traffic. One Apache-2.0 binary: 72 native providers behind an OpenAI-compatible API, or serve vLLM and llama.cpp on your own GPUs. Keys, budgets, guardrails, semantic cache, WAF.

Created Apr 2026

Total54
30 days +0