Normal Accidents: Living with High-Risk Technologies

The Devil’s Advocate™ →Go uptown Air New Zealand Flight TE901 In which the curmudgeonly old sod puts the world to rights. Index — Click ᐅ to expand: The Devil’s Advocate Tell me more Sign up for our newsletter — or just get in touch: for ½ a weekly 🍺 you get to consult JC. Ask about it here.

The Jolly Contrarian’s Dictionary The snippy guide to financial services lingo.™ →Go uptown Index — Click ᐅ to expand: Dictionary Tell me more Sign up for our newsletter — or just get in touch: for ½ a weekly 🍺 you get to consult JC. Ask about it here.

Accident (n.)
An inevitable occurrence due to the action of immutable laws. — Ambrose Bierce, The Devil’s Dictionary

This is one of those “books that will change your life”. Well — that should change lives — that it was written in 1984 — Charles Perrow passed away in 2019 — suggests that, maybe it hasn’t: that the irrationalities that motivate so much of what we do are more pervasive than plainly written common sense.

Charles Perrow was a sociologist who fell into the discipline of systems analysis: analysing how social structures like businesses, governments and public utilities, being loose networks of autonomous individuals, work. Perrow’s focus fell upon organisations that present specific risks to operators, passengers, innocent bystanders — nuclear and other power stations, airways, shipping lines, but the read-across to the financial systems is obvious — where a combination of what he termed complex interactions and tight coupling in distributed systems mean that catastrophic accidents are not just likely but, from time to time, inevitable. Such unpredictable failures are an intrinsic property of a complex, tightly coupled system, not merely a function of “operator error” that can be blamed on a negligent employee — although be assured, that is how management will be inclined to characterise it if given half a chance.

If this is right, it has profound consequences for how we who inhabit complex, tightly-coupled systems, should think about risk.

If you work in financial services, you do inhabit a complex, tightly-coupled system.

It seems unarguably right.

Yet you don’t hear many people talking about how to handle normal accidents.

Complex interactions and tight coupling

First, some definitions.

• Complex interactions: Perrow anticipates the later use of the concept of “complexity” — a topic which is beginning to infuse the advocacy part of this site — without the benefit of systems analysis, since it hadn’t really been invented when he was writing, but to describe interactions between non-adjacent sub-components of a system that were neither intended nor anticipated by the designers of the system. Complex interactions are not only unexpected, but for a period of time (which may be critical, if the interacting components are tightly coupled) will be incomprehensible. This may be because the interactions cannot be seen, buried under second-order control and safety systems, or even because they are not believed. If your — wrong — theory of the game is that the risk in question is a ten sigma event, expected only once in one hundred million years, you may have a hard time believing it could be happening in your fourth year of operation, as the partners of Long Term Capital Management may tell you. Here even epistemology is in play. Interactions that were not in our basic conceptualisation the world, are not ones we can reasonably anticipate. These interactions were, QED, not designed into the system; no one intended them. “They baffle us because we acted in terms of our own designs of a world that we expected to exist—but the world was different.”
• Linear interactions: Contrast complex interactions with much more common “linear interactions”, where parts of the system interact with other components that precede or follow them in the system in ways that are expected and planned: “if this, then that”. In a well-designed system, these will (of course) predominate: any decent system should mainly do what it is designed to do and not act erratically in normal operation. Some systems are more complex than others, but even in the most linear systems are susceptible to some complexity — where they interact with the environment. Cutting back into the language of systems analysis for a moment, consider that linear interactions are a feature of simple and complicated systems, and can be “pre-solved” and brute-force computed; at least in theory. They can be managed by algorithm, or playbook. But complex interactions, by definition, cannot — they are the interactions the algorithm didn’t expect.
• Tight coupling: Complex interactions are only a source of catastrophe if another condition is satisfied: that unexpectedly-interacting components of the complex system are “tightly coupled” — processes happen fast, can’t be turned off, failing components can’t be isolated. Perrow’s observation is that complex systems tend to be more tightly coupled than we realise, and we usually only find out the hard way.

Normal accidents

Where you have a complex system, you should therefore expect accidents — yes, and opportunities, quirks and serendipities, to be sure, but here we are talking about risk — to arise from unexpected, non-linear interactions. Such accidents, says Perrow, are “normal”, not in the sense of being regular or expected, but in the sense that it is an inherent property of the system to have this kind of accident at some point or other.

Are financial systems complex? About as complex as any distributed system known to humankind. Are they tightly coupled? Well, you could ask the principals of LTCM, Enron, Bear Stearns, Amaranth Advisors, Lehman brothers or Northern Rock, if any of those venerable institutions were still around to tell you about it. But yes. Might reckless mortgage securitisation, excess leverage and flash boys have been on Perrow’s mind? We rather think so: “New financial instruments such as derivatives and hedge funds and new techniques such as programmed trading further increase the complexity of interactions. Breaking up a loan on a home into tiny packages and selling them on a world-wide basis increases interdependency.” He wrote this in 1999, for Pete’s sake.

How to deal with system accidents

So, financial services risk controllers take note: if your system is a complex, tightly-coupled system — and it is — you cannot solve for systemic failures. You can’t prevent them. You have to have arrangements in place to deal with them. These arrangements need to be able to deal with the unexpected interactions of components in a complex system, not the predictable effects of a merely complicated one.

Why make the distinction between complex and complicated like this? Because we in the financial services industry are in the swoon of automated, pre-configured safety mechanisms — think chatbots, risk taxonomies, playbooks, checklists, neural networks, even ~ cough ~ contractual rights — and while these may help resolve isolated and expected failures in complicated components, they have no chance of resolving systems failures, which, by definition, will confound them. Instead, these safety mechanisms will get in the way. They are of the system. They are part of what has failed. Not only that: safety mechanisms, by their existence, add complexity in the system — they create their own unexpected interactions — and when a system failure happens they can make it harder to detect what is going on, much less how to stop it.

When Kramer hears about this ...

File:Shit hits fan.jpg

Kramer hearing about this, yesterday.

So far, so hoopy; but here’s the rub: we can make our systems less complex and reduce tight coupling by careful design, functional redundancy and iterative improvement, but, as long as it is a complex system with the scope for complex interaction, we cannot eliminate system accidents altogether. They are, as coders like to joke, a feature, not a bug.

Furthermore, in our efforts to pre-solve for catastophe, we tend not to simplify: to the contrary, we add prepackaged “risk mitigation” components: policies, taxonomies, key performance indicators, tick-boxes, processes, rules, and new-fangled bits of kit to the process in the name of risk management. To be sure, these give our middle management layer comfort; they can set their RAG statuses green, and it may justify their planned evisceration of that cohort of troublesome subject matter experts who tend to foul up the mechanics of the Heath Robinson machine — but who will turn out to be just the people you wish you hadn’t fired when the shit hits the fan.

Here is the folly of elaborate, complicated safety mechanisms: adding components to any complex system increases its complexity. That, in itself, makes dealing with system accidents, when they occur, harder. The safety mechanisms beloved of the middle management layer derive from experience. They secure stables from which horses have bolted. They are, as Jason Fried elegantly put it, “organisational scar tissue. Codified responses to situations that are unlikely to happen again.”

They are, in a word, linear responses to what will be, when it happens, by definition a non-linear problem.

Not only do linear safety mechanisms exacerbate or even create their own accidents, but they also afford a degree of false comfort that encourages managers, who typically have financial targets to meet, not safety ones — to run the system harder, thus increasing the tightness of the coupling between unrelated components. That same Triple A rating that lets your risk officer catch some zeds at the switch encourages your trader to double down. I’m covered. What could go wrong?

Part of the voyeuristic pleasure of Perrow’s book is the salacious detail with which he documents the sequential failures at Three Mile Island, the Space Shuttle Challenger, Air New Zealand’s Erebus crash, among many other disasters and near-misses. The chapter on maritime collisions would be positively hilarious were it not so distressing.

“Operator error” is almost always the wrong answer

Human beings being system components, it is rash to blame them when they are component that is constitutionally disposed to fail — we are frail, mortal, inconstant, narratising beings — even when not put in a position, through system design or economic incentive that makes failure inevitable. A ship’s captain who is expected to work a 48-hour watch and meet unrealistic deadlines is hardly positioned, let alone incentivised to prioritise safety. Perrow calls these “forced operator errors”: “But again, “operator error” is an easy classification to make. What really is at stake is an inherently dangerous working situation where production must keep moving and risk-taking is the price of continued employment.”

If an operator’s role is simply to carry out a tricky but routine part of the system then the inevitable march of technology makes this ever more a fault of design and not personnel: humans, we know, are not good computers. They are good at figuring out what to do when something unexpected happens; making decisions; exercising judgment. But they — we — are lousy at doing repetitive tasks and following instructions. As The Six Million Dollar Man had it, we have the technology. We should damn well use it.

If, on the other hand, the operator’s role is to manage complexity — then technology, checklists and pre-packaged risk taxonomies can only take you so far and, at the limit, can get in the way. Perrow’s account of the control deck at Three Mile Island is instructive:

Besides, about this time—just four or five minutes into the accident—another more pressing problem arose. The reactor coolant pumps that had turned on started thumping and shaking. They could be heard and felt from far away in the control room. Would they withstand the violence they were exposed to? Or should they be shut off? A hasty conference was called, and they were shut off. (It could have been, perhaps should have been, a sign that there were further dangers ahead, since they were “cavitating”—not getting enough emergency coolant going through them to function properly.) In the control room there were three audible alarms sounding, and many of the 1,600 lights (on-off lights and rectangular displays with some code numbers and letters on them) were on or blinking. The operators did not turn off the main audible alarm because it would cancel some of the annunciator lights. The computer was beginning to run far behind schedule; in fact it took some hours before its message that something might be wrong with the PORV finally got its chance to be printed. Radiation alarms were coming on. The control room was filling with experts; later in the day there were about forty people there. The phones were ringing constantly, demanding information the operators did not have. Two hours and twenty minutes after the start of the accident, a new shift came on.

This is, as Perrow sees it, the central dilemma of the complex system. The nature of normal accidents is such that they need experienced, wise operators on the ground ready to think quickly and laterally to solve unfolding problems, but the enormity of the risks involved mean that central management are not prepared to delegate so much responsibility to the mortal, inconstant, narratising meatware.

See also

• Complexity