Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC

Convenient LLMs at Home I’ve been wanting to experiment with LLMs in my homelab, but didn’t want the overhead of a dedicated GPU machine or the slowness of CPU processing. I also wanted everything to be convenient long-term: updates needed to be automated, and if the OS dies rebuilding needed to be quick and easy, etc. Running NixOS with WSL on my gaming PC seemed like the perfect solution, but I kept running into several challenges: Concerns of my vram getting locked to LLMs WSL would shut down automatically; Microsoft doesn’t support WSL running if you aren’t actively using it. NixOS on WSL didn’t support Nvidia out of the box. I refused to manage a separate Ubuntu box that would need reconfiguring from scratch. After hacking away at it for a number of weeks, I’ve now solved the blocks: Ollama (by default) unloads models if they haven’t been used in the past 5 minutes. WSL automatically starts up, and stays running. running. Configured Nvidia Container Toolkit for NixOS on ... Read full article.