GoKawiilIndependent developer trains 3.8B-parameter LLM for under $1,000 on rented B200 GPUs
Hugo Vergnes built a config-driven training framework called little-lm and used it to train a 3.8-billion-parameter language model from scratch on 65 billion tokens, taking 43 hours on eight rented B200 GPUs at a total cost of $998. The model scored 0.384 on the CORE benchmark, outperforming Andrej Karpathy's nanochat d32 model, which cost about the same to train but scored 0.310.