Learning to solve hard problems in RL for LLMs by never giving up

This is a blog post for my recent paper on RL post-training of LLMs: introducing the Matthew Effect and proposing to solve it with Never Give Up. It is presented interactively and less formally, more like how I give the talk. For a deeper, more technical dive, check out the paper on arxiv and code on github.

What is your eval actually measuring? #

Every good RL practitioner has no doubt seen an eval curve go up. Here is the AIME 2025 eval during our RL training of Olmo 3.1 RL-Zero Math (1) (1)see Olmo 3.1 blog post and arxiv

Training Olmo 3 7B base with RL on Dolci RL-Zero math improves its overall math ability. Or does it?

What does this curve really mean?

Our eval is an average over 30 AIME questions. Let’s break those 30 questions down into 3 levels of difficulty. Every question that our initial, pre-RL model gets 0 for pass@32 will be labelled “hard”. The other questions we’ll divide evenly by into “medium” and “easy” based on their pass-rates. So our initial pass@1 averages will be 0%, 3.8%, 22.7% for our subsets. How do you think performance on each subset will evolve?

▶ Play Our AIME evals during Olmo 3.1 RL-Zero split into three levels of difficulty by their initial accuracy. Easy AIME problems improve drastically but problems that start with pass@32=0 mostly end with pass@32=0!

Averaging our AIME eval was hiding something important: the majority of our improvements are coming from the easiest problems going from somewhat solved to mostly solved. The hardest problems are barely improving. This is clearly visible if you look at how each example’s solve rate changes over time (see plot in the margin). ⊕ Accuracy of each AIME eval example over training. We order examples by difficulty from top (initial model pass@32=0) to bottom (initial model pass@1 > 30%). The hardest examples (top rows) barely improve over training. The model mainly learns to better solve easy and medium-difficulty examples that were already reasonably-well solved. We call this discrepancy the Matthew Effect. But this is for math RL on LLMs. What about other domains?

We evaluate code RL and agentic RL using Deepcoder and DeepSWE, two nice open-source projects that released models and logs. We can use the initial model to split each benchmark into difficulty buckets (Deepseek-R1-Distilled-Qwen-14B on LCBv6) or we can use existing task length/difficulty labels (SWEBench).

Initial Model Show me the gains from RL!

... continue reading