Just Ask for Generalization (2021)
Generalizing to what you want may be easier than optimizing directly for what you want. We might even ask for "consciousness". This blog post outlines a key engineering principle I’ve come to believe strongly in for building general AI systems with deep learning. This principle guides my present-day research tastes and day-to-day design choices in building large-scale, general-purpose ML systems. Discoveries around Neural Scaling Laws, unsupervised pretraining on Internet-scale datasets, and o