Categories / More categories

LLM Infrastructure

Running and serving large language models, LLM API gateways and proxies, and retrieval-augmented generation (RAG) systems.

Repos

123

Stars

73,559

Last 30 days

+5,730

winfunc

A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.

Created Jan 2025

ai anthropic anthropic-claude api chain-of-thought
Total5,352
30 days +9
microsoft

AICI: Prompts as (Wasm) Programs

Created Sep 2023

ai inference language-model llm llm-framework
Total2,072
30 days +2
intentee

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

Created Apr 2024

Total1,694
30 days +14
Ai00-X

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.

Created Jul 2023

ai aiagents chatgpt chatgpt-api embed
Total624
30 days +2
automataIA

GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration

Created Sep 2025

Total540
30 days +7
llamastash

Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.

Created May 2026

ai gguf llamacpp llm lmstudio
Total304
30 days +77
arcships

Unified LLM access layer in Rust — one API for 325 AI providers. 统一 LLM 服务接入层:一套 API 接入 325 家 AI 服务商。

Created Jul 2026

Total243
30 days +41
truespar

Native Rust inference server for open models on NVIDIA and Apple Silicon. OpenAI- and Anthropic-compatible APIs, GGUF + safetensors + MLX, FP8/NVFP4/MXFP4/Q8/Q4, built-in Studio

Created Sep 2026

ai aimodels blackwell cuda inference
Total165
30 days +28
lsuen

SUGT - local AI gateway & skills console. Centralize LLM API keys, one OpenAI/Anthropic-compatible proxy for Claude Code, Codex & OpenCode. Homebrew & winget. Fully open source (Apache-2.0).

Created Jul 2026

Total138
30 days +128
noumena-labs

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.

Created Mar 2026

Total125
30 days +3
AlephantAI

Alephant is an open-source AI Agent Gateway for routing, tracking, and controlling LLM usage across AI agents, members, and workflows, and for publishing agent capabilities as paid endpoints with x402 and MPP payment rails.

Created May 2026

agent ai ai-agent ai-gateways claude
Total119
30 days +4
Noveum

Built for demanding AI workflows, this gateway offers low-latency, provider-agnostic access, ensuring your AI applications run smoothly and quickly.

Created Nov 2024

Total100
30 days +2
enapt

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

ai anthropic-api candle chatgpt-alternative claude-code
Total90
30 days +30
ubiomni

One Endpoint, Every Model. Route, Monitor, and Failover — All From Your Terminal.

Created Mar 2026

ai ai-agents model-router
Total86
30 days +0
9j

High-performance AI routing proxy built in Rust with automatic failover, priority-based routing, and support for 15+ providers (Anthropic, OpenAI, Cerebras, Minimax, Kimi, etc.)

Created Nov 2025 Archived

ai anthropic claude-code claude-code-router openai
Total519
30 days +1