Categories / More categories
LLM Infrastructure
Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.
Repos
123
Stars
73,522
Last 30 days
+5,266
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust ๐ฆ
Created Mar 2024
Run, serve, query and edit AI models โ a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Created Oct 2026
A programmable system for high-performance model serving
Created Jan 2025
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
Created May 2026
High-performance KV cache storage for LLM inference โ GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.
Created Jan 2026
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang โ 0.76 ยตs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
Created Aug 2026
Free, open-source app that runs AI chat models on your own computer โ and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
Created Feb 2026
Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-compatible API.
Created Jun 2026
Pure-Rust GGUF inference engine with quantized CPU, Metal & CUDA kernels, MoE support, OpenAI-compatible server. Benchmarked head-to-head against llama.cpp.
Created Aug 2026
Run 70B+ LLMs on Apple Silicon by using SSD as extended memory โ intelligent layer streaming and caching for Mac
Created Feb 2026