Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,589

Last 30 days

+5,854

microsoft

AICI: Prompts as (Wasm) Programs

Created Sep 2023

ai inference language-model llm llm-framework
Total2,072
30 days +3
OrbitInfra

OrbitKV is a High Performance KV Cache System for LLM Inference.

Created Mar 2025

attention kv-cache kv-cache-offloading llm-inference llm-serving
Total284
30 days +10
gpillon

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

Created Sep 2026

ada ampere blackwell coding-agents cuda
Total70
30 days +41