GoKawiilStructural engineers have a physical stamp that they apply to plans they approve, and if a building falls down due to design problems, they’re directly and personally responsible. Software engineers can deploy code that kills hundreds of people without anywhere near that level of accountability. Here’s why I don’t think that’s right, or even sustainable.
They say regulations are written in blood, and it wasn’t until the failure of the St Francis Dam killed around 400 people in 1928 that licensing of civil engineers become a requirement in California. Software design flaws have already taken a much larger toll, and the future promises to make our work even more deadly. Already the Russian invasion of Ukraine has led to autonomous drones having to take software-driven decisions about who to kill, and its clear that code will be a big part of warfare going forward. AI models are also becoming crucial components in safety-critical systems in almost every industry, and while I’m skeptical about the eschatological predictions of them gaining consciousness, any victims won’t care if the cause of a disaster was a malicious Skynet or just incompetent engineering.
So, there’s plenty of blood already, and more to come, so why has software engineering escaped regulation so far? Here are some of the barriers:
It’s a lot easier to start writing code than it is to build a dam. There are an estimated two million software engineers in the US, versus about two hundred thousand civil engineers, and many coders are self-taught without formal qualifications. Gatekeeping who can practice software engineering just won’t work.
Software has no physical location. Bridges are built in one place, so authorities can observe and control what happens very easily. Code and models can be created anywhere in the world and deployed equally globally, so there’s no clear jurisdiction for any country, let alone any way of knowing what software is deployed within its borders.
Software engineering is a deeply collaborative and iterative process. System designs are constantly evolving as we learn more about user needs, and optimize for different goals. A skyscraper has a design that’s agreed before any construction starts, and changes are infrequent and minor enough that they can be checked without slowing down the building process. There’s no practical way to apply the same civil engineering workflow to our industry, we don’t have an easy template we could follow.
Engineers, even at big technology companies, don’t have the authority to make critical safety decisions. If an executive wants to ship self-driving software that has deadly flaws, they can just overrule any objections. Even if you take a stand and quit, the executive can just hire someone else, with no consequences for either management or the new engineer.
Many software projects don’t have a single clear owner. Open source frameworks may have a team of reviewers, and commercial code bases pass through many hands over time. This distribution of authority makes it hard to figure out who is responsible when something goes wrong.
The culture of our industry is highly individualistic. We value moving fast and breaking things, and Silicon Valley has been highly successful at building software companies that now dominate the world, so it’s hard to argue with the success of that ethos. We’ve long benefitted from a perception that we’re underdogs, and society has been broadly willing to forgive any negative consequences of our actions.
Does this all mean that any kind of regulation is unrealistic? I’m confident that we will end up being held to account for the work we do in the long run, and I think the only question is whether we try to take the initiative and start something ourselves, or whether we will miss that opportunity and have rules dictated to us by outsiders. It’s obvious if you look at the bipartisan pushback against data centers that technology companies are losing the indulgence we’re accustomed to receiving from the public. Once the romantic mythology that has protected us in the past loses its power, we’ll be regulated like any other industry.
... continue reading