War story: the hardest bug I ever debugged

When I was on the Google Docs team, we did a weekly bug triage where we’d look for new issues and randomly assign them to teammates to investigate. One week, we had a new top error by a wide margin. It was a fatal error. This means that it prevented the user from editing without reloading. It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints, so we weren’t even sure it was really happening — but if it was happening it would be really bad. It was Chrome-only starting at a specific release. This is less helpful than it sounds, since we often wrote browser-specific Docs bugs that affected only one of Internet Explorer, Firefox, Safari, and Chrome. I tried to repro in dev. This was important for 2 reasons: Rule out Closure Compiler, Docs’ JavaScript compiler at the time. Debugging in unoptimized code is always easier than the alternative. Okay, how do I begin? I crawled through our logs for in ... Read full article.