Structural engineers have a physical stamp that they apply to plans they approve, and if a building falls down due to design problems, they’re directly and personally responsible. Software engineers can deploy code that kills hundreds of people without anywhere near that level of accountability. Here’s why I don’t think that’s right, or even sustainable.

They say regulations are written in blood, and it wasn’t until the failure of the St Francis Dam killed around 400 people in 1928 that licensing of civil engineers become a requirement in California. Software design flaws have already taken a much larger toll, and the future promises to make our work even more deadly. Already the Russian invasion of Ukraine has led to autonomous drones having to take software-driven decisions about who to kill, and its clear that code will be a big part of warfare going forward. AI models are also becoming crucial components in safety-critical systems in almost every industry, and while I’m skeptical about the eschatological predictions of them gaining consciousness, any victims won’t care if the cause of a disaster was a malicious Skynet or just incompetent engineering.

So, there’s plenty of blood already, and more to come, so why has software engineering escaped regulation so far? Here are some of the barriers:

• It’s a lot easier to start writing code than it is to build a dam. There are an estimated two million software engineers in the US, versus about two hundred thousand civil engineers, and many coders are self-taught without formal qualifications. Gatekeeping who can practice software engineering just won’t work.
• Software has no physical location. Bridges are built in one place, so authorities can observe and control what happens very easily. Code and models can be created anywhere in the world and deployed equally globally, so there’s no clear jurisdiction for any country, let alone any way of knowing what software is deployed within its borders.
• Software engineering is a deeply collaborative and iterative process. System designs are constantly evolving as we learn more about user needs, and optimize for different goals. A skyscraper has a design that’s agreed before any construction starts, and changes are infrequent and minor enough that they can be checked without slowing down the building process. There’s no practical way to apply the same civil engineering workflow to our industry, we don’t have an easy template we could follow.
• Engineers, even at big technology companies, don’t have the authority to make critical safety decisions. If an executive wants to ship self-driving software that has deadly flaws, they can just overrule any objections. Even if you take a stand and quit, the executive can just hire someone else, with no consequences for either management or the new engineer.
• Many software projects don’t have a single clear owner. Open source frameworks may have a team of reviewers, and commercial code bases pass through many hands over time. This distribution of authority makes it hard to figure out who is responsible when something goes wrong.
• The culture of our industry is highly individualistic. We value moving fast and breaking things, and Silicon Valley has been highly successful at building software companies that now dominate the world, so it’s hard to argue with the success of that ethos. We’ve long benefitted from a perception that we’re underdogs, and society has been broadly willing to forgive any negative consequences of our actions.

Does this all mean that any kind of regulation is unrealistic? I’m confident that we will end up being held to account for the work we do in the long run, and I think the only question is whether we try to take the initiative and start something ourselves, or whether we will miss that opportunity and have rules dictated to us by outsiders. It’s obvious if you look at the bipartisan pushback against data centers that technology companies are losing the indulgence we’re accustomed to receiving from the public. Once the romantic mythology that has protected us in the past loses its power, we’ll be regulated like any other industry.

I believe we do have a chance to build a framework that reduces risk without strangling innovation, but our window is closing rapidly. Here’s what I’d propose as concrete steps:

• Most US computer science degrees don’t even offer a required ethics component, let alone agreement to a code. Promising to use your powers for good isn’t enforceable, but at least if we made this mandatory for graduation, nobody would be able to say they weren’t warned about their responsibilities.
• We should be able to set up a recognized professional accreditation structure ourselves, without waiting for outside pressure. There doesn’t have to be anything enforced, just being able to promote it as a mark of excellence and something desirable for senior engineering roles could give it a lot of soft power. One possibility would be using the ACM directly, since they already offer so much.
• Instead of trying to boil the ocean by introducing new regulations across the software industry, we can start by pushing sectors that already have strong regulation like health, and transport, to prototype the kinds of rules we’d like to see.
• I want to create a professional norm that software projects should have a single clear owner, who has the power to veto decisions that affect safety, typically the tech lead. I know this works because at Apple every task and issue, no matter how small had a Directly Responsible Individual (DRI). We won’t be able to stop executives from reassigning or firing owners, but we can ensure there are consequences. For example, if an owning engineer can memorialize their objections with reference to a widely-recognized code of ethics, that makes future lawsuits against the company more likely to succeed. Regulation by litigation isn’t my preferred approach, but it is how America operates.
• The aviation industry has ASRS, an anonymous way of reporting safety incidents that NASA then uses the data to identify dangerous patterns and trends. Setting up a similar system for software would be valuable, especially if it is kept anonymous with whistleblower protections, because then we could start to figure out safety guidelines proactively, and based on data.
• Software might be global, but revenue is local. Many large organizations with an interest in safety spend a lot on software. If they demand quality standards for the products they buy, a lot of suppliers will implement them. This will take an explicitly political push, because many of the largest buyers are in the government sector.

I don’t know if this is the right roadmap, but I do know if we don’t start figuring out our own proposals, pretty soon we’ll be at the mercy of rules that people outside the industry come up with. I’ve focused on safety as the most obvious danger to guard against, but software and AI have enormous impacts in all sorts of ways. We can see how much influence social media has politically for example. If Meta decides to suppress anti-ICE messages, I want there to be clear expectations of how the engineers involved in those systems should behave.

I have my personal ideas on the best way forward, but they are almost certainly wrong, or at least could be improved dramatically. Regulations have to be something that’s driven by a wider community and that’s why I recently helped start the Alliance for Principled Tech, which brings together people in tech who think technology can still be built in the public interest, and who are willing to work toward that: engineers, founders, designers, executives, investors. If that sounds like you, please join us.

We know how these systems get built, and which architectural choices quietly decide the direction of our society. It’s urgent for us to come together, to name the rules we want to build by, and the values we want to stand for.