Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,522

Last 30 days

+5,266

chrishayuk

Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.

Created Mar 2026

huggingface inference llm mechanistic-interpretability metal
Total1,193
30 days +25
NIyueeE

DeepSeek网页端API代理,支持OpenAI与Anthropic兼容接口 | OpenAI & Anthropic compatible API proxy for DeepSeek web

Created Mar 2026

api api-proxy claude-api deepseek-api freeapi
Total883
30 days +69
Ai00-X

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.

Created Jul 2023

ai aiagents chatgpt chatgpt-api embed
Total624
30 days +2
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total297
30 days +291
ferrumox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Created Mar 2026

Total191
30 days +2
AprilNEA

Turn AI subscriptions into standard API endpoints. OpenAI & Anthropic compatible gateway — for Amp Code, Cursor, Factory CLI, and more.

Created Feb 2026

ai-gateway amp anthropic-api claude copilot
Total141
30 days +3
azrtydxb

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.

Created Aug 2026

Total109
30 days +0
mxyhi

Local AI API gateway for OpenAI / Gemini / Anthropic. Runs on your machine, keeps tokens counted (SQLite), offers priority-based load balancing, optional OpenAI Chat↔Responses format conversion, and one-click setup for Claude Code / Codex.

Created Jan 2026

api-key-managenment claude-api claude-code codex openai
Total94
30 days +0
enapt

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

ai anthropic-api candle chatgpt-alternative claude-code
Total90
30 days +0
gpillon

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

Created Sep 2026

ada ampere blackwell coding-agents cuda
Total68
30 days +0
pegainfer-project

OpenAI-compatible speech serving in Rust: streaming TTS and full-duplex voice over GPT-Live. 128 concurrent PersonaPlex sessions on one GPU.

Created Sep 2026

cuda full-duplex gpt-live inference-server llm-serving
Total51
30 days +0
tracel-ai

Democratizing large model inference and training on any device.

Created Dec 2024 Archived

burn large-model-inference open-webui openai-api rust
Total267
30 days +0