GitHub topics

gguf

47% of these repos are in LLM Infrastructure.

Repositories

21

Total stars

49,916

Trending

AlexsJones

Hundreds of models & providers. One command to find what runs on your hardware.

Created Feb 2026

gguf llm localai mlx skill
Total39,124
30 days +1,016
scarletkc

A desktop app for the Google Colab CLI: accounts, GPU sessions with cost guards, port forwarding and plugins.

Created Nov 2026

colab desktop-app gguf google-colab gpu
Total297
30 days +296
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total296
30 days +291
llamastash

Zero-overhead, terminal-native local-LLM orchestrator. Run any engine, any model via the same proxy.

Created May 2026

ai gguf llamacpp llm lmstudio
Total303
30 days +77
giannisanni

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Created Jul 2026

Total239
30 days +9
Epistates

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Created Jan 2026

ai ane apple-silicon deep-learning distillation
Total333
30 days +3

Top

AlexsJones

Hundreds of models & providers. One command to find what runs on your hardware.

Created Feb 2026

gguf llm localai mlx skill
Total39,124
Epistates

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Created Jan 2026

ai ane apple-silicon deep-learning distillation
Total333
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total296
ShelbyJenkins

The Easiest Rust Interface for Local LLMs and an Interface for Deterministic Signals from Probabilistic LLM Vibes

Created Dec 2023

ai candle gguf llm rust
Total254
giannisanni

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

Created Jul 2026

Total239
ferrumox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Created Mar 2026

Total191
noumena-labs

AI inference, packed simply. A blazing-fast, zero-dependency WebGPU runtime to run GGUF models directly in the browser. Features a symmetric API for seamless local execution and cloud provider routing. Built with Rust & C++.

Created Mar 2026

Total125
AarambhDevHub

🦀 Decoder-only LLM built from scratch in pure Rust using Candle — no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).

Created Jun 2026

ai-agents candle decoder-only deep-learning dora
Total112
zadazddq

LM Studio: Local LLM Desktop Runtime with Offline Inference and Model Management

Created Nov 2026

desktop-ai gguf lm-studio lm-studio-2026 lm-studio-creative-workflow
Total92
enapt

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Created Feb 2026

ai anthropic-api candle chatgpt-alternative claude-code
Total90
mikecvet

The Natural Language Shell integrates OpenAI's GPTs, Anthropic's Claude, or local GGUF-formatted LLMs directly into the terminal experience, allowing operators to describe their tasks in either POSIX commands or fluent human language

Created Feb 2024

gguf gpt-4 llama2 llm rust
Total73
Dyluhn

Shape-specialized ROCm inference for Qwen3.8 Flash Next on RDNA4

Created Aug 2026

gguf qwen rdna4 rocm speculative-decoding
Total69
ghostapp-ai

The Private Agent OS — search files, run AI agents, connect to 10,000+ tools via the complete protocol stack (MCP, AG-UI, A2UI, A2A). Zero cloud. Zero telemetry. On-device inference.

Created Feb 2026

a2a a2ui ag-ui agent-orchestration agent-os
Total53

Related

All topics →