Categories / LLM Infrastructure

A local LLM server built for concurrent work. Drop-in replacement for Ollama and/or OpenAI and Ollama APIs on one port. Requests that share a prompt reuse each other's KV cache instead of each prefilling it. Rust, wrapping llama.cpp.

Last push 2 months ago

cudaggufinference-enginellama-cppllmlocal-llmmlxollamaollama-apiopenai-apirocmrustvulkan

Created

Mar 2026

Total stars

192

Last 7 days

0

Last 30 days

0

Last 12 months

0

README badge

Rusty Stars badge for ferrumox/fox
[![Rusty Stars](https://rustystars.dev/badge/ferrumox/fox.svg)](https://rustystars.dev/ferrumox/fox)