Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
Categories / More categories
Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.
Repos
123
Stars
73,522
Last 30 days
+5,266
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
DeepSeek网页端API代理,支持OpenAI与Anthropic兼容接口 | OpenAI & Anthropic compatible API proxy for DeepSeek web
Created Mar 2026
Pure Rust Inference Engine
Created May 2026
The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.
Created Jul 2023
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Created Oct 2026
OpenAI compatible API for TensorRT LLM triton backend
Created Nov 2023
A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.
Created Mar 2026
Turn AI subscriptions into standard API endpoints. OpenAI & Anthropic compatible gateway — for Amp Code, Cursor, Factory CLI, and more.
Created Feb 2026
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
Created Aug 2026
Local AI API gateway for OpenAI / Gemini / Anthropic. Runs on your machine, keeps tokens counted (SQLite), offers priority-based load balancing, optional OpenAI Chat↔Responses format conversion, and one-click setup for Claude Code / Codex.
Created Jan 2026
Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
Created Feb 2026
TensorRT-LLM server with Structured Outputs (JSON) built with Rust
Created Oct 2024
An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.
Created Sep 2026
A RAG API server written in Rust following OpenAI specs
Created Apr 2024
OpenAI-compatible speech serving in Rust: streaming TTS and full-duplex voice over GPT-Live. 128 concurrent PersonaPlex sessions on one GPU.
Created Sep 2026
Democratizing large model inference and training on any device.
Created Dec 2024 Archived
A secure and efficient gateway for interacting with OpenAI's API, featuring load balancing, user request handling without individual API keys, and global access control.
Created May 2023 Archived