A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
Categories / More categories
Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.
Repos
123
Stars
73,373
Last 30 days
+0
Top 100 of 123
A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU.
Created Jun 2026
A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.
Created Jan 2025
Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Created May 2026
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
Created Apr 2026
All-in-one platform for search, recommendations, RAG, and analytics offered via API
Created Mar 2023
Local AI routing gateway with Agent client takeover — reuse existing client logins, call 50+ AI clients through one base URL, with unified OpenAI / Anthropic / Images APIs, multi-account routing, check-in, balance and request reports · 本地 AI 路由网关 + Agent 客户端接管:复用各家客户端登录态,一个 base_url 调用 50+ 家模型额度,统一 OpenAI / Anthropic / 生图接口,多账号选路、签到、余额与请求报表一体管理
Created Sep 2026
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Created Apr 2024
Get up and running with GLM-5.3-flash, DeepSeek, Gemma and other open source frontier models.
Created Mar 2026
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
Created Sep 2026
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
Created Mar 2024
This is a project that unifies the management of LLM APIs. It can call multiple backend services through a unified API interface, convert them to the OpenAI format uniformly, and support load balancing. Currently supported backend services include: OpenAI, Anthropic, DeepBricks, OpenRouter, Gemini, Vertex, etc.
Created Jul 2024
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
LLM 逆向工程接口管理 | 通过标准 OpenAI API 访问 ChatGPT / gpt4free / Bard / Claude / HuggingChat / 通义千问 等 AI 的破解版 || ChatGPT reverse engineering API management | Access all reverse engineered LLM libs by standard OpenAI API format || 免费 ChatGPT Free GPT LLM API | 逆向工程 转 OpenAI API | converts all llm libs to OpenAI API
Created Sep 2023
🧠 Motorhead is a memory and information retrieval server for LLMs.
Created Mar 2023
DeepSeek网页端API代理,支持OpenAI与Anthropic兼容接口 | OpenAI & Anthropic compatible API proxy for DeepSeek web
Created Mar 2026
Self-hosted AI API and MCP gateway for organizations: SSO and RBAC, per-user identity for MCP tool calls, PII redaction and tool-call guards, rate limits, budgets, cost accounting and audit logs.
Created Apr 2026
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Created Feb 2026
Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.
Created Oct 2023
Pure Rust Inference Engine
Created May 2026
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
Created Nov 2025
The fastest, lightest, and easiest-to-integrate AI gateway on the market. Fully open-sourced.
Created Apr 2025
The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.
Created Jul 2023
eLLM: Run Long-Horizon Inference Faster on CPUs Than on GPUs
Created Jun 2025
GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration
Created Sep 2025
A high-performance and light-weight router for vLLM large scale deployment
Created Jul 2025
🕷️ The fastest open-source AI Gateway and LLM Gateway — one binary with built-in console to route, secure, budget and observe every LLM call across OpenAI, Anthropic, Gemini, Bedrock and more.
Created Oct 2026
Simple CLI tool for deterministic routing of queries between local and hosted LLM models
Created Jun 2026
Edge full-stack LLM platform. Written in Rust
Created Nov 2023
Blazing-fast LLM inference in pure Rust. No PyTorch and Python runtime.
Created Jun 2025
underclass: an OpenAI-compatible pooling proxy that pins sessions to one account (prompt cache stays warm), cools quota-exhausted subscriptions until their window resets, and fails fast with the earliest Retry-After when the whole pool is dry.
Created Sep 2026
Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.
Created May 2026
Build for agents — from the AI Gateway to the entire application stack.
Created Mar 2026
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Created Oct 2026
OrbitKV is a High Performance KV Cache System for LLM Inference.
Created Mar 2025
基于Rust编写的中转API破甲项目,支持Responses / Chat Completions / Anthropic Messages API,支持ChatGPT/Claude/Grok/Gemini/Deepseek/Glm/Kimi等大多数主流模型
Created Oct 2026
A programmable system for high-performance model serving
Created Jan 2025
SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.
Created Jul 2026
Local-first AI memory layer for any LLM. Persistent knowledge graph, entity extraction, semantic retrieval. Works with Ollama, OpenAI, Anthropic, or any OpenAI-compatible backend.
Created Jun 2026
The execution control plane for AI agents. Govern every model request, MCP tool call, A2A delegation, and downstream action—while keeping your native protocols intact: six native LLM protocols in, six native LLM protocols out.
Created May 2026
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
Created May 2026
High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.
Created Jan 2026
OpenAI compatible API for TensorRT LLM triton backend
Created Nov 2023
Open-source Rust TemporalStore for LLM context management, temporal features, and risk control
Created Jun 2026
A model-agnostic GPU runtime for shipping models as verified programs.
Created Aug 2026
A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.
Created Mar 2026
WaLiAPI 是一款本地运行的 LLM API 网关桌面软件(Tauri 2 + React + Rust)。支持各类 LLM 厂商渠道接入、Ollama 自部署模型、CPA 方式 ChatGPT 渠道接入,统一支持 Chat Completions / Responses / Anthropic Messages 三协议转换使用。内置安全审计引擎、知识库 RAG(HNSW + FTS5 混合检索)、Wiki 知识引擎和 MCP Server(29 个工具),配合 Codex、Claude Code、Gemini CLI 等 AI 编程工具使用。
Created Jul 2026
CC-Gateway-Pro: AI Provider Gateway with Vision Routing & Project-Level Provider Binding (forked from cc-switch)
Created May 2026
Production-grade RAG API built in Rust. Hybrid search with HNSW dense vectors and BM25 sparse matching, cross-encoder reranking, layout-aware document extraction via Docling, and 94.5% accuracy on Open RAG Bench. Powered by Cerebras, Groq, Milvus, and Jina AI.
Created Feb 2026
Created Feb 2024
SUGT - local AI gateway & skills console. Centralize LLM API keys, one OpenAI/Anthropic-compatible proxy for Claude Code, Codex & OpenCode. Homebrew & winget. Fully open source (Apache-2.0).
Created Jul 2026
AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.
Created Mar 2026
Self-hosted Rust LLM gateway with OpenAI-compatible APIs, load balancing, failover, and a reusable Rust kernel.
Created Jul 2025
Alephant is an open-source AI Agent Gateway for routing, tracking, and controlling LLM usage across AI agents, members, and workflows, and for publishing agent capabilities as paid endpoints with x402 and MPP payment rails.
Created May 2026
A high-performance AI gateway written in Rust. It sits in front of your LLM backends and exposes a unified API surface to clients, handling routing, load balancing, rate limiting, and observability.
Created Jun 2026
Onboard AI tools in one step, and catch the rest. Relay sets up Claude Desktop, Claude Code, and Codex to run through your LiteLLM AI Gateway with zero dev setup and routes shadow AI (Notion AI, Perplexity, …) there too.
Created Jul 2026
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
Created Aug 2026
Fully open-source, high-performance LLM inference engine for the RK3588 NPU
Created Oct 2026
Built for demanding AI workflows, this gateway offers low-latency, provider-agnostic access, ensuring your AI applications run smoothly and quickly.
Created Nov 2024
Local AI API gateway for OpenAI / Gemini / Anthropic. Runs on your machine, keeps tokens counted (SQLite), offers priority-based load balancing, optional OpenAI Chat↔Responses format conversion, and one-click setup for Claude Code / Codex.
Created Jan 2026
The open engine for running AI on your own computer. Fast, exact, open source.
Created Nov 2026
Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
Created Feb 2026
OpenAI compatible API servers for the Qwen3 TTS models
Created Jan 2026
Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-compatible API.
Created Jun 2026
Serve your models fast on your own infra in 3 steps.
Created Oct 2026
AI API gateway with protocol translation, inline moderation, streaming parity, and local moderation training.
Created Nov 2025
TensorRT-LLM server with Structured Outputs (JSON) built with Rust
Created Oct 2024
Pure-Rust GGUF inference engine with quantized CPU, Metal & CUDA kernels, MoE support, OpenAI-compatible server. Benchmarked head-to-head against llama.cpp.
Created Aug 2026
An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.
Created Sep 2026
A physics-grounded, cost-aware optimization loop for vLLM
Created Mar 2026