GoKawiilSoftware engineer warns AI agents inherit 'bad priors' from non-expert training feedback
An experienced software engineer argues that AI agents perform well in domains their operators understand deeply, but operators are blindly trusting model judgment in countless other areas they cannot personally evaluate. The author points to 'slop'—technically functional but poor-quality code patterns—as evidence that models were rewarded during training by non-experts, embedding flawed defaults into the model's behavior.