A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
Categories / More categories
Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.
Repos
123
Stars
73,589
Last 30 days
+5,854
A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
Pure Rust Inference Engine
Created May 2026
OrbitKV is a High Performance KV Cache System for LLM Inference.
Created Mar 2025
A programmable system for high-performance model serving
Created Jan 2025
A model-agnostic GPU runtime for shipping models as verified programs.
Created Aug 2026
Pure-Rust GGUF inference engine with quantized CPU, Metal & CUDA kernels, MoE support, OpenAI-compatible server. Benchmarked head-to-head against llama.cpp.
Created Aug 2026
An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.
Created Sep 2026
Run any model on Intel silicon
Created May 2026