I trained a small transformer in 1.5hrs and it beats many LLMs

I trained a small transformer from scratch in 1.5hrs on a 5090

Beats many LLMs, and scores the same as TRM/HRM

This is an upgrade to my previous model

Faster, better, cheaper and still open source.

Also gets 7% on ARC-2

Discussion on Twitter, Code on github

Performance on ARC-1 public eval. I only compare against models that do similar test time training

This is the 3rd blog in a series of works on ARC-AGI. Prev: Blog 2, Blog 1.

Many ppl thought the prev result was impossible. It got attention from top researchers and went viral on X. Eg: Discussions by Lucas Beyer, Jeremy Howard, Rohan Anil, and comments by many others.

Why work on this?

... continue reading