Photo by Scott Wills

Before you start worrying I've gone all Icke on you the answer is no, but what's interesting about the question is that it's a lot harder to answer than it used to be. Todd Vernon's article on the still-unexplained stock market crash of May 3rd reminded me of something, but it took a while to put my finger on it. Then I realized, it was from Vernor Vinge's A Deepness in the Sky.

In the novel Vinge describes financial weapons that are unleashed by a malevolent intelligence on an unsuspecting civilization, business plans that explode their systems. At the time I was naive enough that I couldn't picture how just transmitting an idea could have that sort of impact, but now it's pretty obvious. All you need is an alien in the early 2000's to email an investment banker "Hey, everyone wants triple-A securities. If you bundle this risky debt into tranches you can persuade the ratings agencies to certify part of it as AAA and you'll make a fortune".

What I find fascinating as a computer engineer is that this is all my profession's fault. We've built an amazing infrastructure of information plumbing to allow the financial system to operate on auto-pilot, without those pesky humans sucking up salaries and slowing the whole process down. The late lamented Tanta opened my eyes to how the job of a mortgage underwriter had changed from being a gatekeeper, almost a detective trying to sniff out risky loans, to someone whose focus was making sure that the submitted forms matched the criteria set in the computer programs they now used to grade loans. In a very literal sense these programs are artificial intelligences, expert systems that try to mechanize the thought process that we used to rely on loan officers to go through.

These loans were then bundled and sold as securities based on complex computer models based on historical data (which didn't include falling house prices). These securities were then bought and resold by investors using their own complex trading programs. This was all in the debt market, but the same takeover of decisions by software occurred in equities, and the end result is a system composed of hundreds of thousands of different programs all talking to each other and making enormous financial decisions on their owner's behalf. What's truly scary about this size of system is that it's reached a scale where it's so complex that it's impossible to understand why anything happened. That's what worries me about the May 3rd crash, it's the first time a vital information system we've built has proved too complex to debug. In the pre-computer world, you could just interview everyone who bought or sold on the daya of a crash and ask them why they took action. We can't do that with our programs, which is why the crash will remain a mystery.

In another part of Deepness, Vinge describes a civilization whose systems have reached the point where they're so entangled and baroque that nobody can fix them when they crash, and the whole world is destined to collapse. Again I couldn't picture that when I first read it, but now I can. In the financial crisis we had cargoes that couldn't be shipped because the standard letters of credit between banks were no longer being honored. Our fundamental mechanisms for delivering food were taken out by problems in our financial information systems! Our AIs don't need to become self-aware to turn into Skynet, they're quite capable of causing serious damage just as they are.

So how can we fix this? As a start we need all programs making financial decisions to produce a clear audit trail in a standard format, explaining not only what actions they took but why. With this sort of log available for every market participant, forensic investigators would be able to build a picture of the cause of events like May 3rd. It may well prove to be impossible for some existing code to come up with meaningful reasons for its decision tree, but that's a feature of this proposal, not a bug. If the operators can't justify the actions of the programs they're running, then that's a clear sign they're too dangerous to be interfacing with our financial system.