A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.
Created Jan 2025
Categories / More categories
Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.
Repos
123
Stars
73,559
Last 30 days
+5,730
A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.
Created Jan 2025
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
Created Apr 2026
All-in-one platform for search, recommendations, RAG, and analytics offered via API
Created Mar 2023
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Created Apr 2024
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
Created Nov 2025
The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.
Created Jul 2023
GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration
Created Sep 2025
🕷️ The fastest open-source AI Gateway and LLM Gateway — one binary with built-in console to route, secure, budget and observe every LLM call across OpenAI, Anthropic, Gemini, Bedrock and more.
Created Oct 2026
Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.
Created May 2026
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Created Oct 2026
SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.
Created Jul 2026
High-scale LLM gateway, written in Rust. OpenTelemetry-based observability included
Created Oct 2024
The execution control plane for AI agents. Govern every model request, MCP tool call, A2A delegation, and downstream action—while keeping your native protocols intact: six native LLM protocols in, six native LLM protocols out.
Created May 2026
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
Created May 2026
High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.
Created Jan 2026
OpenAI compatible API for TensorRT LLM triton backend
Created Nov 2023
Open-source Rust TemporalStore for LLM context management, temporal features, and risk control
Created Jun 2026
A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.
Created Mar 2026
Turn AI subscriptions into standard API endpoints. OpenAI & Anthropic compatible gateway — for Amp Code, Cursor, Factory CLI, and more.
Created Feb 2026
Production-grade RAG API built in Rust. Hybrid search with HNSW dense vectors and BM25 sparse matching, cross-encoder reranking, layout-aware document extraction via Docling, and 94.5% accuracy on Open RAG Bench. Powered by Cerebras, Groq, Milvus, and Jina AI.
Created Feb 2026
SUGT - local AI gateway & skills console. Centralize LLM API keys, one OpenAI/Anthropic-compatible proxy for Claude Code, Codex & OpenCode. Homebrew & winget. Fully open source (Apache-2.0).
Created Jul 2026
Self-hosted Rust LLM gateway with OpenAI-compatible APIs, load balancing, failover, and a reusable Rust kernel.
Created Jul 2025
Alephant is an open-source AI Agent Gateway for routing, tracking, and controlling LLM usage across AI agents, members, and workflows, and for publishing agent capabilities as paid endpoints with x402 and MPP payment rails.
Created May 2026
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
Created Aug 2026
Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
Created Feb 2026
Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-compatible API.
Created Jun 2026
An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.
Created Sep 2026
Run 70B+ LLMs on Apple Silicon by using SSD as extended memory — intelligent layer streaming and caching for Mac
Created Feb 2026
An open source, Native AI Gateway and LLM proxy built in Rust
Created Mar 2026
🚀 100% local RAG system with one-command setup. Your data never leaves your server.
Created Nov 2025
Open source Enterprise AI Gateway for API, MCP and agent, and AI model traffic. One Apache-2.0 binary: 72 native providers behind an OpenAI-compatible API, or serve vLLM and llama.cpp on your own GPUs. Keys, budgets, guardrails, semantic cache, WAF.
Created Apr 2026