Researchers warn of ‘catastrophic overtraining’ in Large Language Models

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More A new academic study challenges a core assumption in the development of large language models (LLMs), warning that more pre-training data may not always lead to better models. Researchers from some of the leading computer science institutions in the West and around the world — including Carnegie Mellon University, Stanford University, Harvard University, and Princeton University — have introduced the concept of “Catastrophic Overtraining,” showing that extended pre-training can actually make language models harder to fine-tune, ultimately degrading their performance. The study, titled “Overtrained Language Models Are Harder to Fine-Tune”, is available on arXiv and led by Jacob Mitchell Springer, along with co-authors Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. The law of diminishing returns The ... Read full article.