GoKawiilOops: Twenty-eight years ago, Pixar nearly lost 90% of Toy Story 2's digital files – not because a system crashed, but because someone ran a routine Unix command that engineers had been using for years without a second thought. The command /bin/rm -r -f instructs the system to recursively delete everything under a directory without asking for confirmation. At Pixar in 1998, it was apparently executed in the wrong location.
The studio's animation pipeline at the time ran across a network of Unix and Linux machines holding hundreds of thousands of production files. Artists and technical staff had broad access to both personal workspaces and shared production directories. The setup made collaboration easier, but meant a routine cleanup command could reach critical files if issued from the wrong directory.
According to people familiar with the incident, the command propagated beyond its intended scope and began erasing core production data. Oren Jacob, an associate technical director on the film, watched it happen in real time. Files began disappearing from his screen, first individual assets, then entire characters and sequences.
Within moments, roughly 90% of the movie was gone.
"You don't often watch a company vaporize in front of your eyes," Jacob told The Wall Street Journal.
The immediate response was to contain the damage. An emergency call went out to shut down the system, cutting off the deletion mid-process. That worked. The fallback plan did not.
Pixar's backup system was designed precisely for this kind of scenario, but it had been failing silently. Nobody realized it until they tried to use it. "The mechanism we had in place specifically to help us recover from data failures had itself failed," Pixar co-founder Ed Catmull later wrote in Creativity, Inc.
What remained of the film was fragmented and inconsistent. Reconstructing it from pieces would have been slow and uncertain, and the timeline was already tight. Pixar had recently committed to releasing Toy Story 2 theatrically rather than as a direct-to-video sequel, raising both the creative bar and the financial stakes. A long delay could have had serious consequences for the company.
To Pixar's leadership, identifying who triggered the error was beside the point. "Looking for someone to blame doesn't help us learn from mistakes," Catmull said. "We understood that the deletion of the movie was an accident because somebody typed in a command when they were in the wrong directory. We don't know who typed the command, or if they even knew that they were the one who did it, but it didn't matter." What mattered was whether the film could be recovered at all.
The answer turned out to be sitting off-site. Galyn Susman, the film's supervising technical director, had been maintaining a working copy of the project on a machine at her home. She had built a remote workflow during her pregnancy, periodically syncing updated versions of the film so she could continue working after hours. It wasn't part of Pixar's formal backup infrastructure, but it was current enough to matter.
... continue reading