Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,522

Last 30 days

+5,266

microsoft

AICI: Prompts as (Wasm) Programs

Created Sep 2023

ai inference language-model llm llm-framework
Total2,072
30 days +2
chrishayuk

Run, serve, query and edit AI models โ€” a Rust model runtime built on VINDEX3, an executable, inspectable model format.

Created Mar 2026

huggingface inference llm mechanistic-interpretability metal
Total1,193
30 days +25
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total297
30 days +291
novitalabs

High-performance KV cache storage for LLM inference โ€” GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.

Created Jan 2026

inference kv-cache llm vllm
Total221
30 days +3
truespar

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

Created Sep 2026

ai aimodels blackwell cuda inference
Total165
30 days +28
noumena-labs

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.

Created Mar 2026

Total125
30 days +0
azrtydxb

The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang โ€” 0.76 ยตs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.

Created Aug 2026

Total109
30 days +0
enapt

Free, open-source app that runs AI chat models on your own computer โ€” and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

ai anthropic-api candle chatgpt-alternative claude-code
Total90
30 days +0
midagedev

Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in llama-server.

Created Sep 2026

Total78
30 days +0