Repositories
37
Total stars
20,959
Trending
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Created Oct 2026
Fast ML inference & training for ONNX models in Rust
Created Nov 2022
TypeDB: Built for systems, not records
Created Jul 2016
High-performance Ultralytics YOLO inference in Rust with ONNX Runtime, GPU backends, CLI, and WebGPU/WASM.
Created Nov 2025
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
A programmable system for high-performance model serving
Created Jan 2025
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
Created Mar 2024
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
Created May 2026
Top
TypeDB: Built for systems, not records
Created Jul 2016
Fast ML inference & training for ONNX models in Rust
Created Nov 2022
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
Created Mar 2024
Run, serve, query and edit AI models — a Rust model runtime built on VINDEX3, an executable, inspectable model format.
Created Mar 2026
An open-source computer vision framework to build and deploy apps in minutes
Created Jul 2023
Rust wrapper for Microsoft's ONNX Runtime (version 1.8)
Created Aug 2020
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Created Oct 2026
OramaCore is the complete runtime you need for your projects, answer engines, copilots, and search. It includes a fully-fledged full-text search engine, vector database, LLM interface, and many more utilities.
Created Oct 2024
A programmable system for high-performance model serving
Created Jan 2025
High-performance Ultralytics YOLO inference in Rust with ONNX Runtime, GPU backends, CLI, and WebGPU/WASM.
Created Nov 2025
Camelid: a Rust-native local inference backend with evidence-gated model compatibility.
Created May 2026
High-performance KV cache storage for LLM inference — GPU offloading, SSD caching, and cross-node sharing via RDMA. Works with vLLM and SGLang.
Created Jan 2026
A DuckDB extension for in-database inference
Created Sep 2025
LLaMa 7b with CUDA acceleration implemented in rust. Minimal GPU memory needed!
Created Apr 2023
The lowest-overhead LLM router. Production-ready, highly available, one OpenAI-compatible endpoint in front of 80 providers and your own vLLM/SGLang — 0.76 µs per request, no I/O on the request path, cache-affinity routing, RBAC, budgets and a 13-screen UI in the binary.
Created Aug 2026
Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.
Created Feb 2026
Distributed LLM inference fabric for heterogeneous local clusters. Features zero-copy SPSC ring buffers, automated tuning, and an OpenAI-compatible API.
Created Jun 2026
A neural network inference library, written in Rust.
Created Sep 2023
Pure-Rust GGUF inference engine with quantized CPU, Metal & CUDA kernels, MoE support, OpenAI-compatible server. Benchmarked head-to-head against llama.cpp.
Created Aug 2026
Run 70B+ LLMs on Apple Silicon by using SSD as extended memory — intelligent layer streaming and caching for Mac
Created Feb 2026
A CLI tool that generates JSON Typedef schemas from example data
Created Apr 2020
EdgeInfer enables efficient edge intelligence by running small AI models, including embeddings and OnnxModels, on resource-constrained devices like Android, iOS, or MCUs for real-time decision-making. EdgeInfer 旨在资源受限的设备上运行小型 AI 模型(包括向量化和 Onnx 模型),如 Android、iOS 或 MCUs,实现高效的边缘智能,用于实时决策。
Created Nov 2023