A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
50% of these repos are in LLM Infrastructure, 10% in Artificial Intelligence.
Repositories
20
Total stars
35,592
A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
RDNA-native LLM inference engine in Rust.
Created Mar 2026
A model-agnostic GPU runtime for shipping models as verified programs.
Created Aug 2026
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
Created Aug 2021
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Created Aug 2025
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
Created Jul 2024
Pure Rust Inference Engine
Created May 2026
A programmable system for high-performance model serving
Created Jan 2025
A Datacenter Scale Distributed Inference Serving Framework
Created Mar 2025
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
Created Jul 2024
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Created Aug 2025
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
Created Aug 2021
RDNA-native LLM inference engine in Rust.
Created Mar 2026
Pure Rust Inference Engine
Created May 2026
PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.
Created Jan 2026
OrbitKV is a High Performance KV Cache System for LLM Inference.
Created Mar 2025
A programmable system for high-performance model serving
Created Jan 2025
A model-agnostic GPU runtime for shipping models as verified programs.
Created Aug 2026
Lossless compression for LLM weights and KV cache, and the inference engine that runs on it.
Created Sep 2026
Zihuan is a high-performance, highly customizable Rust-based AI Agent platform. Zihuan Impls a interesting Workspace Agent (zihuan code for coding) and an IM Agent (QQ bots, etc.)
Created Nov 2025
Pure-Rust GGUF inference engine with quantized CPU, Metal & CUDA kernels, MoE support, OpenAI-compatible server. Benchmarked head-to-head against llama.cpp.
Created Aug 2026
An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.
Created Sep 2026
Reduce Claude Code token usage by 35-50% when using CJK (Chinese, Japanese, and Korean)
Created Jan 2026
Rust implementation of Attention Residuals from MoonshotAI/Kimi
Created Mar 2026
Run any model on Intel silicon
Created May 2026