GoKawiilMy local model setup on an M4 Pro Mac mini
I run a local LLM server on my M4 Pro Mac mini with 48 GB of RAM. It handles everything from my Hermes agent backend to quick chat queries on my phone. The whole thing takes about 30 minutes to set up.
Here is the stack:
Qwen3.6-35B-A3B-OptiQ-4bit : my main model for anything that needs reasoning or depth
: my main model for anything that needs reasoning or depth Gemma-4-E4B-it-OptiQ-4bit : lightweight model for simple chats, formatting, and other routine tasks
: lightweight model for simple chats, formatting, and other routine tasks oMLX : the inference server
: the inference server Tailscale: tailnet connecting the Mac mini, my iPhone, and my MacBook
Hermes runs as the agent backend on the Mac mini, with my MacBook running the desktop client and my phone running Telegram. For non-Hermes usage I use Apollo on iOS for quick chats (reads like Claude, good for throwaway questions), Pi as my coding agent (I already wrote about that setup), and Raycast AI on my Mac for random things.
The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it. I was regularly maxing out two $200/month subscriptions and it felt like I was getting different things from them at different points. Sometimes a model was fine, sometimes it degraded with no notice.
Data privacy is another issue. You do not know what these companies do with your data once they have it. They might limit how it gets used, they might sell it, they might expose it. Either way, it creates an operational security risk. If you work with sensitive code, client data, or proprietary workflows, sending it to a third-party API is a decision you make once and cannot undo.
... continue reading