Why This Matters
This breakthrough demonstrates that small, efficient transformers can achieve competitive performance on complex benchmarks like ARC-AGI-1 with minimal training data and cost, challenging the notion that large models are necessary for high performance. It highlights the potential for more accessible, cost-effective AI development, fostering innovation and democratization in the industry.
Key Takeaways
- Small transformers can achieve high scores on ARC-AGI-1 with minimal training time and data.
- Focus on sample efficiency can significantly reduce AI development costs and barriers.
- Open-source models and methods promote wider participation and faster innovation in AI research.
I trained a small transformer from scratch in 1.5hrs on a 5090
Beats many LLMs, and scores the same as TRM/HRM
This is an upgrade to my previous model
Faster, better, cheaper and still open source.
Also gets 7% on ARC-2
Discussion on Twitter, Code on github
Performance on ARC-1 public eval. I only compare against models that do similar test time training
This is the 3rd blog in a series of works on ARC-AGI. Prev: Blog 2, Blog 1.
Many ppl thought the prev result was impossible. It got attention from top researchers and went viral on X. Eg: Discussions by Lucas Beyer, Jeremy Howard, Rohan Anil, and comments by many others.
Why work on this?
... continue reading
GoKawiil