GoKawiilA follow-up question on a 50,000 token conversation took three to five minutes before the first token appeared. Not the full answer. The first token. That is not a chatbot, it is a batch job, and you go and make a cup of coffee while it thinks.
To the dozen other geeks obsessed with maximising your Mac Studio, I come with good tidings. By good tidings I mean I spent three weeks debugging a cache leak so you don’t have to.
This started with antirez/ds4, a brilliant experiment in running large models on consumer hardware. Two months ago I impulse-bought an M3 Mac Studio Ultra. I feared RAM prices would make the 96GB configs unattainable soon (spoiler: they did), and I really wanted to dive into local inference stacks whilst I am on parental leave. The goal was simple: run a frontier model on a single machine, keep it warm, and have a conversational AI that actually understands context.
It turned out to be more complicated than that. The model fits in unified memory and answers coherently, But after weeks of daily use I made a hard pivot: I dropped DS4 Flash and switched to Qwen 3.5 122B. Two separate things then happened, and it is worth keeping them apart. The model swap was a speed-and-fit decision. Making the new model actually usable meant fixing three bugs in my own serving stack that had nothing to do with which model I ran.
Why I switched models #
DS4 Flash is a genuinely good model and antirez’s stack is a brilliant piece of work. It just was not the right model for what I do. This is a fit story, not a fault story.
My use case is long-context agentic coding: pair programming where the model holds thousands of tokens of conversation, code, and tool output, and I need near-instant turns to stay in the flow. For that specific workflow the prefill latency was the dealbreaker. Past 50k tokens a simple follow-up took three to five minutes before the first token appeared, and you cannot pair program with a model that makes you wait for a cup of tea. By the time it caught up I had already moved on to the next problem. That is not DS4 being slow in some absolute sense, it is a mismatch between how that setup handles long-context prefill on my hardware and what my workflow needs.
Qwen 3.5 122B looked like a better fit for the M3 Ultra on every axis I cared about:
Near frontier, fully local. No API calls, no rate limits, no data leaving the machine. A 122B model that rivals proprietary systems, running entirely offline.
No API calls, no rate limits, no data leaving the machine. A 122B model that rivals proprietary systems, running entirely offline. The right active-param size for the Ultra’s bandwidth. The M3 Ultra’s memory bandwidth is large. A 122B MoE with roughly 10B active params sits in the sweet spot where the GPU can feed the compute units without stalling.
... continue reading