GitHub topics

local-llm

34% of these repos are in LLM Infrastructure, 26% in AI Agents and Coding Assistants.

Repositories

23

Total stars

63,726

Trending

RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total296
30 days +291
tonhowtf

A local MCP server that gives your coding agent a desktop toolbox: downloads from YouTube, torrents and every site yt-dlp supports, media conversion, files, chats and system cleanup. One line connects Claude Code, Codex, Cursor or Gemini CLI, and sensitive actions wait for your click. Free and open source for Windows, macOS and Linux.

Created Feb 2026

Total14,507
30 days +165
gavamedia

Run full Kimi K3 on a single device. And an OpenAI-compatible API server for local chat and coding agents.

Created Jul 2026

Total860
30 days +11
giannisanni

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Created Jul 2026

Total239
30 days +9

Top

tonhowtf

A local MCP server that gives your coding agent a desktop toolbox: downloads from YouTube, torrents and every site yt-dlp supports, media conversion, files, chats and system cleanup. One line connects Claude Code, Codex, Cursor or Gemini CLI, and sensitive actions wait for your click. Free and open source for Windows, macOS and Linux.

Created Feb 2026

Total14,507
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total296
giannisanni

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Created Jul 2026

Total239
ferrumox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Created Mar 2026

Total191
BigStrongSun

CCswitch的全面升级,专门优化codex体验。支持官方订阅和第三方模型混用,自动路由,支持v2subagent使用第三方模型

Created Aug 2026

ai-tools ccswitchmulti codex codex-multirouter deepseek
Total134
oxide-lab

Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant

Created Aug 2025

ai-chat artificial-intelligence candle chatbot conversational-ai
Total119
zadazddq

LM Studio: Local LLM Desktop Runtime with Offline Inference and Model Management

Created Nov 2026

desktop-ai gguf lm-studio lm-studio-2026 lm-studio-creative-workflow
Total92
enapt

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

ai anthropic-api candle chatgpt-alternative claude-code
Total90
gpillon

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

Created Sep 2026

ada ampere blackwell coding-agents cuda
Total67
bug-ops

A memory-first AI agent that remembers why decisions were made — not just the last message. Runs local (Ollama), cloud (Claude · OpenAI · Gemini), or decentralized TEE. Graph memory, self-learning skills, multi-model routing, sandboxed tools. MCP · ACP · A2A. One Rust binary.

Created Feb 2026

Total66
yarenty

You already have a local model. Kowalski gives it jobs: hordes of small steps where code does the fetching, SQL and file writing, and the model only writes a query or a paragraph. One binary, Ollama or your own key. Rust, MIT.

Created Mar 2025

Total66

Related

All topics →