Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,373

Last 30 days

+0

Top 100 of 123

ai-dynamo

A Datacenter Scale Distributed Inference Serving Framework

Created Mar 2025

diffusion disaggregated-serving kubernetes llm-inference omni
Total8,527
30 days +0
magnitudedev

Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU.

Created Jun 2026

Total7,913
30 days +0
winfunc

A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.

Created Jan 2025

ai anthropic anthropic-claude api chain-of-thought
Total5,353
30 days +0
NVIDIA-NeMo

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Created May 2026

Total3,658
30 days +0
openlake-project

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

Created Apr 2026

blackwell gpt gpu high-performance llm
Total3,092
30 days +0
devflowinc

All-in-one platform for search, recommendations, RAG, and analytics offered via API

Created Mar 2023

actix actix-web ai artificial-intelligence diesel
Total2,727
30 days +0
aimod-cc

Local AI routing gateway with Agent client takeover — reuse existing client logins, call 50+ AI clients through one base URL, with unified OpenAI / Anthropic / Images APIs, multi-account routing, check-in, balance and request reports · 本地 AI 路由网关 + Agent 客户端接管:复用各家客户端登录态,一个 base_url 调用 50+ 家模型额度,统一 OpenAI / Anthropic / 生图接口,多账号选路、签到、余额与请求报表一体管理

Created Sep 2026

accio autoclaw catpaw cline docker
Total2,546
30 days +0
microsoft

AICI: Prompts as (Wasm) Programs

Created Sep 2023

ai inference language-model llm llm-framework
Total2,073
30 days +0
intentee

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Created Apr 2024

ai llamacpp llm llmops load-balancer
Total1,695
30 days +0
LlamaEdge

The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

Created Oct 2023

Total1,666
30 days +0
RunanywhereAI

Get up and running with GLM-5.3-flash, DeepSeek, Gemma and other open source frontier models.

Created Mar 2026

deepseek frontier-ai glm-5-3 llms local-ai-infrastructure
Total1,571
30 days +0
ollaya-dev

Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

Created Sep 2026

calibration decision-model decision-models inference-server jev
Total1,474
30 days +0
StarlightSearch

Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀

Created Mar 2024

ai cloud generative-ai hacktoberfest high-performance
Total1,332
30 days +0
Xerxes-2

High Performance LLM Reverse Proxy

Created Mar 2025

Total1,311
30 days +0
yym68686

This is a project that unifies the management of LLM APIs. It can call multiple backend services through a unified API interface, convert them to the OpenAI format uniformly, and support load balancing. Currently supported backend services include: OpenAI, Anthropic, DeepBricks, OpenRouter, Gemini, Vertex, etc.

Created Jul 2024

aws azure claude gemini grok
Total1,274
30 days +0
chrishayuk

Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.

Created Mar 2026

huggingface inference llm mechanistic-interpretability metal
Total1,190
30 days +0
perplexityai

Perplexity open source garden for inference technology

Created Nov 2025

Total958
30 days +0
RockChinQ

LLM 逆向工程接口管理 | 通过标准 OpenAI API 访问 ChatGPT / gpt4free / Bard / Claude / HuggingChat / 通义千问 等 AI 的破解版 || ChatGPT reverse engineering API management | Access all reverse engineered LLM libs by standard OpenAI API format || 免费 ChatGPT Free GPT LLM API | 逆向工程 转 OpenAI API | converts all llm libs to OpenAI API

Created Sep 2023

bard chatbot chatgpt claude free
Total953
30 days +0
getmetal

🧠 Motorhead is a memory and information retrieval server for LLMs.

Created Mar 2023

llmops llms machine-learning ml mlops
Total916
30 days +0
NIyueeE

DeepSeek网页端API代理,支持OpenAI与Anthropic兼容接口 | OpenAI & Anthropic compatible API proxy for DeepSeek web

Created Mar 2026

api api-proxy claude-api deepseek-api freeapi
Total881
30 days +0
gavamedia

Run full Kimi K3 on a single device. And an OpenAI-compatible API server for local chat and coding agents.

Created Jul 2026

kimi kimi-k3 local-ai local-llm
Total860
30 days +0
ThinkWatchProject

Self-hosted AI API and MCP gateway for organizations: SSO and RBAC, per-user identity for MCP tool calls, PII redaction and tool-call guards, rate limits, budgets, cost accounting and audit logs.

Created Apr 2026

ai-gateway ai-governance ai-proxy audit-log audit-logging
Total817
30 days +0
pegainfer-project

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Created Feb 2026

cuda cuda-kernels deepseek gpu inference
Total748
30 days +0
EricLBuehler

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

Created Oct 2023

Total742
30 days +0
Avarok-Cybersecurity

Pure Rust Inference Engine

Created May 2026

cuda dgx dgx-spark gb10 llm-inference
Total737
30 days +0
smg-project

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Created Nov 2025

anthropic anthropic-api chat claude gemini
Total732
30 days +0
t8

Run models too big for your Mac's memory

Created Mar 2026

Total670
30 days +0
Helicone

The fastest, lightest, and easiest-to-integrate AI gateway on the market. Fully open-sourced.

Created Apr 2025

ai cache gateway llm llms
Total647
30 days +0
Ai00-X

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.

Created Jul 2023

ai aiagents chatgpt chatgpt-api embed
Total624
30 days +0
lucienhuangfu

eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs

Created Jun 2025

llm-infernece cpu-inference qwen long-horizon rust-llm
Total602
30 days +0
automataIA

GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration

Created Sep 2025

ai embeddings entity-extraction graphrag knowledge-graph
Total540
30 days +0
vllm-project

A high-performance and light-weight router for vLLM large scale deployment

Created Jul 2025

Total526
30 days +0
eliasbiondo

🕷️ The fastest open-source AI Gateway and LLM Gateway — one binary with built-in console to route, secure, budget and observe every LLM call across OpenAI, Anthropic, Gemini, Bedrock and more.

Created Oct 2026

ai-gateway anthropic guardrails llm llm-gateway
Total468
30 days +0
asdecided

Simple CLI tool for deterministic routing of queries between local and hosted LLM models

Created Jun 2026

Total421
30 days +0
guoqingbao

Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.

Created Jun 2025

agent llm qwen rust vllm
Total356
30 days +0
ghuntley

underclass: an OpenAI-compatible pooling proxy that pins sessions to one account (prompt cache stays warm), cools quota-exhausted subscriptions until their window resets, and fails fast with the earliest Retry-After when the whole pool is dry.

Created Sep 2026

openai openai-proxy-load-balancer
Total303
30 days +0
llamastash

Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.

Created May 2026

ai gguf llamacpp llm lmstudio
Total302
30 days +0
LeenHawk

gproxy is a Rust-based multi-channel LLM proxy that exposes OpenAI / Claude / Gemini-style APIs through a unified gateway, with a built-in admin console, user/key management, and request/usage auditing.

Created Feb 2026

claude gemini gpt llm-proxy
Total296
30 days +0
taichuy

Build for agents — from the AI Gateway to the entire application stack.

Created Mar 2026

ai-gateway claude-api llm-gateway llm-orchestration self-hosted
Total295
30 days +0
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

cuda gguf inference llama-cpp llm
Total289
30 days +0
santiagomed

LLM Orchestrator built in Rust

Created Sep 2023

Total286
30 days +0
OrbitInfra

OrbitKV is a High Performance KV Cache System for LLM Inference.

Created Mar 2025

rust llm-inference kv-cache attention memory-management
Total284
30 days +0
baehyunsol

git-like rag pipeline

Created Oct 2024

Total259
30 days +0
mizaawa

基于Rust编写的中转API破甲项目,支持Responses / Chat Completions / Anthropic Messages API,支持ChatGPT/Claude/Grok/Gemini/Deepseek/Glm/Kimi等大多数主流模型

Created Oct 2026

Total253
30 days +0
basecompute

Fastest LLM inference runtime for Apple Silicon

Created Jun 2026

Total250
30 days +0
pie-project

A programmable system for high-performance model serving

Created Jan 2025

inference inference-engine llm-inference machine-learning mlsystem
Total244
30 days +0
arcships

Unified LLM access layer in Rust — one API for 325 AI providers. 统一 LLM 服务接入层:一套 API 接入 325 家 AI 服务商。

Created Jul 2026

ai c flutter go java
Total243
30 days +0
giannisanni

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Created Jul 2026

cuda gguf glm inference-engine llm
Total239
30 days +0
zaydmulani09

Local-first AI memory layer for any LLM. Persistent knowledge graph, entity extraction, semantic retrieval. Works with Ollama, OpenAI, Anthropic, or any OpenAI-compatible backend.

Created Jun 2026

Total237
30 days +0
traceloop

High-scale LLM gateway, written in Rust. OpenTelemetry-based observability included

Created Oct 2024

artificial-intelligence datascience generative-ai llm llmops
Total233
30 days +0
GetBusbar

The execution control plane for AI agents. Govern every model request, MCP tool call, A2A delegation, and downstream action—while keeping your native protocols intact: six native LLM protocols in, six native LLM protocols out.

Created May 2026

ai-gateway anthropic bedrock circuit-breaker claude
Total227
30 days +0
timtoole02

Camelid: a Rust-native local inference backend with evidence-gated model compatibility.

Created May 2026

apple-silicon gguf inference llama llm
Total221
30 days +0
novitalabs

High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Created Jan 2026

inference kv-cache llm vllm
Total221
30 days +0
npuichigo

OpenAI compatible API for TensorRT LLM triton backend

Created Nov 2023

langchain llm openai-api tensorrt-llm triton-inference-server
Total221
30 days +0
AmineDiro

OpenAI compatible API for serving LLAMA-2 model

Created Aug 2023

Total214
30 days +0
matrixarkai

Open-source Rust TemporalStore for LLM context management, temporal features, and risk control

Created Jun 2026

feature-store matrixark open-source rust storage
Total212
30 days +0
pegainfer-project

A model-agnostic GPU runtime for shipping models as verified programs.

Created Aug 2026

cuda gpu-inference llm-inference rust
Total211
30 days +0
mos1128

多Agent兼容LLM智能网关与效率工具

Created Jan 2026

Total204
30 days +0
ferrumox

Run Qwen 3.8 MAX (2.4T MoE) on a 50GB-RAM consumer machine in pure Rust, experts streamed from disk. Tiny engine, immense model.

Created Jul 2026

Total193
30 days +0
ferrumox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Created Mar 2026

cuda gguf inference-engine llama-cpp llm
Total192
30 days +0
fuzhengwei

WaLiAPI 是一款本地运行的 LLM API 网关桌面软件(Tauri 2 + React + Rust)。支持各类 LLM 厂商渠道接入、Ollama 自部署模型、CPA 方式 ChatGPT 渠道接入,统一支持 Chat Completions / Responses / Anthropic Messages 三协议转换使用。内置安全审计引擎、知识库 RAG(HNSW + FTS5 混合检索)、Wiki 知识引擎和 MCP Server(29 个工具),配合 Codex、Claude Code、Gemini CLI 等 AI 编程工具使用。

Created Jul 2026

Total170
30 days +0
truespar

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

Created Sep 2026

ai aimodels inference inference-engine inference-optimization
Total165
30 days +0
andyzpb

Run DeepSeek-v4-pro (1600B MoE) on a 24GB-RAM consumer machine — rust, streamed from disk.

Created Jul 2026

Total164
30 days +0
tiylabs

TiyGate is a lightweight gateway for highly available LLM services. 一款致力于提供高可用 LLM 服务的轻量级网关。

Created Jun 2026

ai-gateway llm-gateway llm-logs llm-proxy multi-provider
Total148
30 days +0
KeaneFeng

CC-Gateway-Pro: AI Provider Gateway with Vision Routing & Project-Level Provider Binding (forked from cc-switch)

Created May 2026

Total143
30 days +0
RustyRAG

Production-grade RAG API built in Rust. Hybrid search with HNSW dense vectors and BM25 sparse matching, cross-encoder reranking, layout-aware document extraction via Docling, and 94.5% accuracy on Open RAG Bench. Powered by Cerebras, Groq, Milvus, and Jina AI.

Created Feb 2026

actix-web groq llm milvus open-source
Total142
30 days +0
AprilNEA

Turn AI subscriptions into standard API endpoints. OpenAI & Anthropic compatible gateway — for Amp Code, Cursor, Factory CLI, and more.

Created Feb 2026

ai-gateway amp anthropic-api claude copilot
Total141
30 days +0
AtomaAI

Fast serverless LLM inference, in Rust.

Created Jun 2024

Total135
30 days +0
Chleba

High-performance LLM backends proxy with per-user fair-share queuing, round-robin scheduling, and a real-time TUI dashboard. Built in Rust.

Created Feb 2026

fair-share llm-ops message-queue ollama openai-compatible
Total132
30 days +0
lsuen

SUGT - local AI gateway & skills console. Centralize LLM API keys, one OpenAI/Anthropic-compatible proxy for Claude Code, Codex & OpenCode. Homebrew & winget. Fully open source (Apache-2.0).

Created Jul 2026

ai ai-gateway anthropic claude-code codex
Total129
30 days +0
noumena-labs

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.

Created Mar 2026

ai cpp gguf llm local-ai
Total125
30 days +0
majiayu000

Self-hosted Rust LLM gateway with OpenAI-compatible APIs, load balancing, failover, and a reusable Rust kernel.

Created Jul 2025

ai-gateway anthropic api-client async-rust aws-bedrock
Total124
30 days +0
AlephantAI

Alephant is an open-source AI Agent Gateway for routing, tracking, and controlling LLM usage across AI agents, members, and workflows, and for publishing agent capabilities as paid endpoints with x402 and MPP payment rails.

Created May 2026

ai ai-agent ai-gateways claude cost-optimization
Total119
30 days +0
modelpointer

A high-performance AI gateway written in Rust. It sits in front of your LLM backends and exposes a unified API surface to clients, handling routing, load balancing, rate limiting, and observability.

Created Jun 2026

Total115
30 days +0
LiteLLM-Labs

Onboard AI tools in one step, and catch the rest. Relay sets up Claude Desktop, Claude Code, and Codex to run through your LiteLLM AI Gateway with zero dev setup and routes shadow AI (Notion AI, Perplexity, …) there too.

Created Jul 2026

ai-gateway ai-governance ai-monitoring ai-proxy shadow-ai
Total112
30 days +0
huggingface

Testbed for LLM inference with cutile-rs.

Created Mar 2026

Total111
30 days +0
azrtydxb

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.

Created Aug 2026

ai-gateway inference litellm llm llm-gateway
Total109
30 days +0
lxe
lxe / dsx

DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) on 2x RTX 3090 + 128 GB RAM. Rust + CUDA, T16 quantization. 40+ tok/s measured with expert deferral + FP8 projections.

Created Oct 2026

Total107
30 days +0
DJAndries

A protocol and modular application suite for language models.

Created Mar 2023

Total105
30 days +0
Noveum

Built for demanding AI workflows, this gateway offers low-latency, provider-agnostic access, ensuring your AI applications run smoothly and quickly.

Created Nov 2024

ai ai-gateway api gateway
Total100
30 days +0
mxyhi

Local AI API gateway for OpenAI / Gemini / Anthropic. Runs on your machine, keeps tokens counted (SQLite), offers priority-based load balancing, optional OpenAI Chat↔Responses format conversion, and one-click setup for Claude Code / Codex.

Created Jan 2026

claude-code codex openai openai-api opencode
Total94
30 days +0
Veloresearch

The open engine for running AI on your own computer. Fast, exact, open source.

Created Nov 2026

cuda gguf inference llama-cpp llm
Total91
30 days +0
enapt

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

anthropic-api candle decentralized distributed-inference gguf
Total90
30 days +0
ubiomni

One Endpoint, Every Model. Route, Monitor, and Failover — All From Your Terminal.

Created Mar 2026

ai ai-agents model-router
Total86
30 days +0
rwilliamspbg-ops

Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-compatible API.

Created Jun 2026

af-xdp computer-networks deep-learning distributed-computing gpu-computing
Total85
30 days +0
meshapplied

Serve your models fast on your own infra in 3 steps.

Created Oct 2026

gpu inference llm nvfp4 nvidia
Total80
30 days +0
edgee-ai

The official Rust library for the Edgee AI Gateway

Created Apr 2024

edge rust edgee edge-computing ai
Total80
30 days +0
ai-switch

Aides大模型本地网关,Ai服务商快捷切换,AI本地路由,AI厂商聚合。

Created Jul 2026

cc-switch claude codex cpa grok
Total79
30 days +0
CassiopeiaCode

AI API gateway with protocol translation, inline moderation, streaming parity, and local moderation training.

Created Nov 2025

Total79
30 days +0
midagedev

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

Created Sep 2026

cuda deepseek gguf glm inference
Total78
30 days +0
guidance-ai

TensorRT-LLM server with Structured Outputs (JSON) built with Rust

Created Oct 2024

guidance openai-api tensorrt-llm cfg json
Total73
30 days +0
antonellof

Pure-Rust GGUF inference engine with quantized CPU, Metal & CUDA kernels, MoE support, OpenAI-compatible server. Benchmarked head-to-head against llama.cpp.

Created Aug 2026

ai inference inference-engine llm-inference rust
Total72
30 days +0
Dyluhn

Shape-specialized ROCm inference for Qwen3.8 Flash Next on RDNA4

Created Aug 2026

gguf qwen rdna4 rocm speculative-decoding
Total69
30 days +0
gpillon

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

Created Sep 2026

blackwell coding-agents cuda inference-engine llm
Total67
30 days +0
jungledesh

A physics-grounded, cost-aware optimization loop for vLLM

Created Mar 2026

cli inference-optimization infra rust vllm
Total67
30 days +0