Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience

I built a system that lets LLMs automatically learn and improve problem-solving strategies over time, inspired by Andrej Karpathy's idea of a "third paradigm" for LLM learning. The basic idea: instead of using static system prompts, the LLM builds up a database of strategies that actually work for different problem types. When you give it a new problem, it selects the most relevant strategies, applies them, then evaluates how well they worked and refines them. For example, after seeing enough word problems, it learned this strategy: 1) Read carefully and identify unknowns, 2) Define variables with units, 3) Write equations, 4) Solve step-by-step, 5) Verify the answer. All strategies are stored as human-readable JSON that you can inspect and edit. I tested it on math benchmarks and saw decent improvements - 8.6% better on Arena Hard, 6.67% on AIME24. After 500 queries, the system had created 129 strategies and refined 97 of them. The implementation is an open-source plugin for ... Read full article.