GoKawiilNPC-Forge and TERMy development
I love research and development, you may have heard of me because of PJON (Padded Jittering Operative Network). It is a network protocol I started developing in 2010, which was recently implemented in silicon by the ETH Zurich university thanks to the research of Pius Sieber.
I had a chance to focus for 2 months on my personal projects since early July, during the strange times of AI price hikes and the end of subsidized tokenmaxing. I was curious to see if I could develop from scratch a terminal assistant capable of handling simple natural language requests. I have a bad memory and got used to ask to copilot "activate the virtual environment" or similar trivial operations spending a non negligible sum every month. I started thinking, maybe I can do something to make my workflow more efficient? Do I really need trillions of parameters to accomplish those tasks?
Transformers at home
I started an open-ended research on the feasibility of implementing a generative model at home and training it from scratch on the computer I used to play Kerbal Space Program in the early 2010s "upgraded" with 16GB of RAM, NVIDIA GTX 1050 Ti (4GB VRAM) and a i7-4790K (4.0GHz 8 cores) CPU. In my experiments I tend to look for minimalism, so I imposed myself a constrained environment to be forced to work towards an elegant and efficient solution.
I first developed a framework to train and evaluate transformers, which I implemented from scratch in Python. I have started with something very similar to NanoGPT with 100-200M parameters, then I added flash attention, and all the expected optimizations, I even tried novel architectures like Mamba. The results were generally unsatisfactory, creepy if not outright scary, like the following:
Enter your prompt (or leave blank for empty start): What is an alien? Generating... Using tokenizer: /ollm/checkpoints/gpt/tinyostrich/tokenizer.model === Generated Text === He's not a member of the world. He can't believe anything anymore. All of those animals are looking like excrements, but every mouth is not a bad one. They look alike, all of which are really terrible. They get their own life, and each is a sign that they're not. They're all right, they say. They're just some different things they can make. They've never seen them. They were better not. Some of them've got their rights. Some of them have to go. But we're not looking for the future on how many. The last two of them are all in the universe.
(the word excrements was used instead of an expletive composed of 4 letters that I am not willing to publish in here)
All my tests produced models that were prone to enter in loops repeating the same sentence, and even if trained on QA they were rarely able to consistently answer questions, specially if technical. I trained models on a collection of royalty free books from the Project Gutemberg, a lot of open-source software, and various datasets available on Huggingface.
I quickly understood that this approach was not feasible; a proper run would have required at least a month of training non-stop. I was amazed by how my models looked alive and magical, but I was also ashamed because they were incredibly wasteful and effectively useless.
... continue reading