GoKawiilRun huge mixture-of-experts models from a swarm of peers, with the colibri engine. Pure C, no dependencies.
One machine shares a model. Any other machine chats with it. Nothing is downloaded up front: the bytes an inference actually touches arrive from a peer on first use and stay in a local mirror, so the second question is served from local disk at full speed. The engine binary is never modified.
Any machine may join, GPU or not. The engine was built for CPU and SSD first; a GPU only makes it faster, never different, and the output is byte-for-byte the same either way. Networks that pool GPUs recruit from the few. lumabri recruits from everyone.
Quick start
make
On the machine that has a model (any colibri model directory):
./lumabri serve --model /path/to/model
On a machine that wants to chat (it needs a colibri build for the engine):
./lumabri chat --tracker < server-ip > :7300 --engines-dir /path/to/colibri/c
That is all. The first answer is slower while the working set crosses the network. Afterwards the mirror in ~/.lumabri keeps serving even if the server goes offline.
... continue reading