GoKawiilEssay argues no equivalent 'commons' exists for posttraining LLM data
A new essay examines how large language model pretraining relied on an enormous, freely available body of text—Wikipedia, Common Crawl, ArXiv, Stack Exchange, Project Gutenberg and open-source code—that had accumulated over decades. The author notes that Common Crawl alone holds roughly 10 petabytes of archived web data, and that models like GPT-3 and the original Llama were built largely from this material. The piece then argues that no comparable public resource exists for the posttraining stage of building AI models.