GoKawiilI’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch.
The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wanted to understand inference at a level deeper than “run the binary,” and I wanted a project where every performance claim had to be earned against a real, well-known baseline rather than asserted.
What Ferrox actually is
At its core, Ferrox loads a GGUF file — the same quantized model format llama.cpp uses — and runs inference on it. Two ways to use it:
A CLI , ferrox , with llama.cpp-compatible flags. Point it at a model, get a completion.
, , with llama.cpp-compatible flags. Point it at a model, get a completion. A server, ferrox-server , that speaks the OpenAI chat-completions API. Anything built against ChatGPT’s API — a chat UI, an agent framework, a test harness — works against it unchanged, just pointed at localhost .
Under the hood, model weights are memory-mapped straight off disk and never fully decompressed into RAM — dequantization happens fused into the dot product, at the moment the weight is actually needed. That’s the same trick llama.cpp uses, and it’s a big part of why both engines can run an 8B-parameter model on a laptop with a few gigabytes of memory instead of thirty.
Try it: build, download a GGUF, run
Ferrox does not ship weights. You download a local .gguf the same way you would for llama.cpp — I recommend the Hugging Face CLI ( pip install -U huggingface_hub ):
git clone https://github.com/antonellof/ferrox.git cd ferrox cargo build --release -p ferrox-cli -p ferrox-server --features metal mkdir -p models # ~1.2 GB smoke test hf download TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \ tinyllama-1.1b-chat-v1.0.Q8_0.gguf --local-dir models # Optional: small instruct chat model (~0.8 GB) hf download bartowski/Llama-3.2-1B-Instruct-GGUF \ Llama-3.2-1B-Instruct-Q4_K_M.gguf --local-dir models
... continue reading