So, this blog post will be about LLMs, and everyone has opinions about that. To be upfront about it, I’m a skeptic (bordering on hater), yet I like experimenting with stuff so I download and run them locally on my Mac. And I’ll teach you how to do it too, if you’d like!
Some call them fancy autocomplete, some argue that they are sentient and should have rights. The truth is somewhere in between. Yes, they perform next word prediction, but it’s so complex that there’s nontrivial emergent behavior. No, they don’t have creativity or a mind. I believe one day we can create sentient machines, but not in this current iteration, maybe not before we go extinct.
Now that we’re out of the science fiction territory, let’s talk about their strengths. Laurie has a great post about it, which I highly recommend, but in summary they are generally good at summarizing text, regurgitating home maintenance advice from reddit, or telling you that you have cancer.
I also use them for brain-dumping. I find it hard to keep a journal, because I find it boring, but when you’re pretending to be writing to someone, it’s easier. If you have friends, that’s much better, but some topics are too personal and a friend may not be available at 4 AM.
I mostly ignore its responses, because it’s for me to unload, not to listen to a machine spew slop. I suggest you do the same, because we’re anthropomorphization machines and I’d rather not experience AI psychosis. It’s better if you don’t give it a chance to convince you it’s real. I could use a system prompt so it doesn’t follow up with dumb questions (or “YoU’Re AbSoLuTeLy CoRrEcT”s), but I never bothered as I already don’t read it.
Lastly, I’m interested in them because it’s tech and I like tech. I don’t believe they make you that much productive, and I never let them write for me. When I ask it something, I always fact-check, they are known to hallucinate (or bullshit, if you prefer) and I’ve experienced this too many times to trust it. Most importantly, just avoid asking questions that can’t be easily verified, there’s enough fake news around anyway.
You may be asking, “Well I can do everything you said with ChatGPT free tier, why bother running them locally?” There are a few reasons, and I’ve already mentioned them briefly:
I like experimenting with things. It’s marvelous that you can download this 12 GB file and your computer talks to you marginally better than ELIZA! Joking aside, we accepted the concept of LLMs too quickly, when the truth is that we never expected computers to figure out human speech before robots were walking among us. So it feels a bit magical when my computer has better grammar than I do, and I can summon one whenever I want without a supercomputer. People have secrets and some secrets shouldn’t leave your computer. Companies are not well equipped to keep your sensitive data, and current trends show that they can retain your data or use it for training. I feel like a local LLM is a better choice for these use cases. I don’t feel comfortable giving money to the AI companies. Every single one of them has done some kind of unethical thing, and the way the AI race is going, a hypothetical ethical one will be left behind. They intentionally hype AI, destroy the environment, and plagiarize people’s hard work. I’m thankful for the open-weight models they provide and will keep using them, luckily they can’t take that away from me.
If you still want to run an LLM on your macOS, let’s get started. There are two options that I recommend, one is open-source and the other is easier to use, as all things are. I only cover macOS because that’s what I have; if that’s not your platform, you can still follow this guide with platform-specific tweaks, or find another guide.
It’s created by Georgi Gerganov and works really well with a ton of configuration options. It supports a lot of platforms, can download models, and has a basic web UI for convenience. You can install it with Nix using the following command: nix profile install nixpkgs#llama-cpp . For other installation methods, check their instructions.
... continue reading