GitHub topics

llama-cpp

31% of these repos are in LLM Infrastructure, 10% in Desktop Apps.

Repositories

19

Total stars

6,068

Trending

RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total294
30 days +286
scarletkc

A desktop app for the Google Colab CLI: accounts, GPU sessions with cost guards, port forwarding and plugins.

Created Nov 2026

colab desktop-app gguf google-colab gpu
Total296
30 days +281
ollaya-dev

Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

Created Sep 2026

calibration decision-model decision-models inference-server jev
Total1,474
30 days +124
timoncool

Free offline AI video dubbing studio for Windows — voice cloning, translation, subtitles & on-screen-text localization. 100% local, one native .exe, zero Python.

Created Jun 2026

ai ai-dubbing ai-video asr dubbing
Total297
30 days +94
lucasjinreal

A Pure Rust based LLM, VLM, VLA, TTS, OCR Inference Engine, powering by Candle & Rust. Alternate to your llama.cpp but much more simpler and cleaner..

Created Mar 2025

audio8-tts jev jev-act kokoro-tts llama-cpp
Total533
30 days +22
automataIA

GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration

Created Sep 2025

Total540
30 days +7

Top

ollaya-dev

Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

Created Sep 2026

calibration decision-model decision-models inference-server jev
Total1,474
automataIA

GraphRAG-rs is a high-performance, state-of-the-art Rust implementation of GraphRAG (Graph-based Retrieval Augmented Generation) that builds knowledge graphs from documents and enables natural language querying with configurable entity extraction and local LLM integration

Created Sep 2025

Total540
lucasjinreal

A Pure Rust based LLM, VLM, VLA, TTS, OCR Inference Engine, powering by Candle & Rust. Alternate to your llama.cpp but much more simpler and cleaner..

Created Mar 2025

audio8-tts jev jev-act kokoro-tts llama-cpp
Total533
timoncool

Free offline AI video dubbing studio for Windows — voice cloning, translation, subtitles & on-screen-text localization. 100% local, one native .exe, zero Python.

Created Jun 2026

ai ai-dubbing ai-video asr dubbing
Total297
RustingStudio

Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.

Created Oct 2026

anthropic-api cuda gguf inference llama-cpp
Total294
ferrumox

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Created Mar 2026

Total191
DingoOz

Live truecolor terminal dashboard for a locally running LLM: tok/s, TTFT, GPUs, context, MTP acceptance, layer and real expert routing (llama.cpp)

Created Sep 2026

gpu-monitoring llama-cpp llm mixture-of-experts ratatui
Total178

Related

All topics →