Normal Accidents: Living with High-Risk Technologies: Difference between revisions

← Older edit

Normal Accidents: Living with High-Risk Technologies (view source)

Revision as of 13:01, 14 April 2023

7,271 bytes added , 14 April 2023

no edit summary

Amwelladmin

Bureaucrats, Interface administrators, Administrators

82,891

edits

@@ Line 1: / Line 1: @@
-{{a|devil|
+{{a|systems|{{image|Erebus|gif|Air New Zealand Flight TE901}}}}{{quote|'''[[Accident]]''' /ˈaksɪd(ə)nt/ ''(n).'' <br>
-[[File:Erebus.gif|450px|frameless|center|Air New Zealand Flight TE901]]
-}}{{quote|{{d|Accident|/ˈaksɪd(ə)nt/|n|}}
 An inevitable occurrence due to the action of immutable laws.
 :— {{author|Ambrose Bierce}}, {{br|The Devil’s Dictionary}}}}
+{{quote|Humans in general do not reason well (even experts can be found to make simple mistakes in probabilities and interpretation of evidence); heroic effort would be needed to educate the general public in the skills needed to decide the complex issues of risk.
+:— {{author|Charles Perrow}}, ''Normal Accidents'', Chapter 9}}
 This is one of those “books that will change your life”. Well — that ''should'' change lives — that it was written in 1984 — {{author|Charles Perrow}} passed away in 2019 — and it isn’t on the bookshelf of every [[thought leader]] in the land suggests that, maybe it hasn’t: that the irrationalities that motivate so much of what we do are more pervasive than plainly written common sense.
 {{author|Charles Perrow}} was a sociologist who fell into the discipline of [[systems analysis]]: analysing how social structures like businesses, governments and public utilities, being loose networks of autonomous individuals, work. Perrow’s focus fell upon organisations that present specific risks to operators, passengers, innocent bystanders — nuclear and other power stations, airways, shipping lines: the read-across to the financial systems is obvious — where a combination of what he termed '''[[complexity|complex interactions]]''' and '''[[tight coupling]]''' in distributed systems mean that catastrophic accidents are not just likely but, from time to time, ''inevitable''. Such unpredictable failures are an intrinsic property of a complex, tightly coupled system, not merely a function of “operator error” that can be blamed on a negligent employee — although be assured, that is how management will be [[inclined]] to characterise it if given half a chance.
+The classic case of such a tightly-coupled system is a nuclear power plant. Perrow was an accident investigator at the Three Mile Island incident. The early part of his book contains a fascinating blow-by-blow account of how TMI unfolded and how close it came to being catastrophically worse than it was.
+Yet, while there were no fatalities, it is premature to conclude that the technology is therefore safe.
+{{Quote|“Large nuclear plants of 1,000 or so megawatts have not been operating very long—only about thirty-five to forty years of operating experience exists, and that constitutes “industrial infancy” for complicated, poorly understood transformation systems.”}}
+The unnerving practical conclusion that Perrow draws is that, for all the easy speeches<ref name="syed">[https://www.thetimes.co.uk/article/f8a262f8-4490-11ec-b414-b1f6389ab345 We are too emotional about risk — no wonder we make bad decisions]— Matthew Syed, ''The Sunday Times'', 14 November 2021.</ref> given about the relative low risk of nuclear power compared with traditional fossil fuel-based energy generation, it is just far too early to draw meaningful conclusions about the tail risk of nuclear meltdown. It is like rolling a die six times, and concluding that, because a six has not yet come up, one is not possible.
+The potential for unanticipatable accidents that trigger unstoppable catastrophic chain reactions is incalculable, and the time horizon over which these accidents could occur or have effect is literally millennial. Which traditional industries these risks are better understood and generally less prevalent.
+To claim that the statistics we have suggest nuclear power is is safe<ref name="syed"/> is to mistake an “absence of evidence” for “evidence of absence”.
+===Financial services relevance===
+This site is mostly concerned with financial services and not nuclear energy, of course. You would think [[financial services]] meet exactly the conditions of [[non-linearity]] and [[tight coupling]] this that Perrow describes.
 If this is right, it has profound consequences for how we who inhabit [[complex]], [[tightly-coupled]] systems, should think about risk. If you work in [[financial services]], you ''do'' inhabit a complex, tightly-coupled system, and it seems unarguably right.
@@ Line 38: / Line 54: @@
 ===When Kramer hears about this ...===
-[[File:Shit hits fan.jpg|300px|thumb|right|Kramer hearing about this, yesterday.]]
+[[File:Shit fan.jpg|400px|thumb|right|Kramer hearing about this, yesterday.]]
-So far, so hoopy; but here’s the rub: we can make our systems less complex and ''reduce'' [[tight coupling]] by careful design, functional redundancy and iterative improvement — air transport has become progressively safer as it has developed: it has learned from each accident — but, as long as it is a complex system with the scope for complex interaction, ''we cannot eliminate [[system accident]]s altogether''. They are, as coders like to joke, a feature, not a bug.
+So far, so hoopy; but here’s the rub: we can make our systems less complex and ''reduce'' [[tight coupling]] by careful design, functional redundancy and iterative improvement — [[air crash|air transport has become progressively safer]] as it has developed: it has learned from each accident — but, as long as it is a complex system with the scope for complex interaction, ''we cannot eliminate [[system accident]]s altogether''. They are, as coders like to joke, a feature, not a bug.
 Furthermore, in our efforts to pre-solve for catastrophe, we tend ''not'' to simplify, but to complicate: we add prepackaged “risk mitigation” components: [[Policy|policies]], [[taxonomy|taxonomies]], [[key performance indicator]]s, [[tick-boxes]], [[dialog box]]es, [[bloatware]] processes, rules, and [[Chatbot|new-fangled bits of kit]] to the process ''in the name of programmatic risk management''.
-These might give the [[middle management]] layer comfort; they can set their [[RAG status]]es green, and it may justify their planned evisceration of that cohort of troublesome [[subject matter expert]]s who tend to foul up the mechanics of the [[Heath Robinson machine]] — but who will turn out to be just the people you wish you hadn’t fired ''when the shit hits the fan''.
+These might give the [[middle management]] layer comfort; they can set their [[RAG status]]es green, and it may justify their planned evisceration of that cohort of troublesome [[subject matter expert]]s who tend to foul up the mechanics of the [[Heath Robinson machine]] — but who will turn out to be just the people you wish you hadn’t fired {{shitfan}}.
 Here is the folly of elaborate, [[complicated]] safety mechanisms: adding components to any complex system ''increases'' its complexity. That, in itself, makes dealing with [[system accident]]s, when they occur, ''harder''. The safety mechanisms beloved of the [[middle management]] layer derive from experience. They secure stables from which horses have bolted. They are, as {{author|Jason Fried}} elegantly put it,
@@ Line 65: / Line 81: @@
 This is, as Perrow sees it, the central dilemma of the [[complex system]]. The nature of [[normal accidents]] is such that they need experienced, wise operators on the ground ready to think quickly and laterally to solve unfolding problems, but the enormity of the risks involved mean that central management are not prepared to delegate so much responsibility to the mortal, inconstant, narratising [[meatware]].
+=== How best to manage? ===
+The optimal means of managing differs depending on the type of risk.
+For non-linear, tightly coupled systems, like banks, this presents a control paradox: complex systems demand decentralised control and local, on-the ground expertise, to react quickly and wisely to unexpected events; tightly-coupled systems that are susceptible to chain reactions require centralised management to control the event quickly at any point in the organisation.
+===What is to be done===
+Dumb operators aren’t the problem, but neither are those perennial culprits: technology, capitalism and greed.
+Technology generally doesn’t ''create'' system accidents so much as fail to stop them and, at the limit, make them harder to foresee and deal with. And there is no imperative, beyond those of scale and economy, which are both very human imperatives to cut corners to profitability — that forces technology upon us. We choose it. We can complain about Twitter all we like, but — yeah.<ref>Twitter isn’t, of course a technology company. It’s a publisher.</ref>
+And while capitalism does generate externalities, unreasonably concentrate economic power, and reward those who have wealth out of all proportion to their contribution, a “capitalist” is no worse at this than a socialist one. (Perrow was writing in 1984, where the distinction between “capitalist” and “socialist” economies was a good deal starker, and the social democratic third way had not really made itself felt. It is a curious irony that we ''feel'' ever more polarised now, whilst our political economies are far more homogenised. Even China, that last socialist standing, is closer to the centre than it was).
+{| class="wikitable"
+|+Suitability of centralisation or local control to management of different systems
+!
+!
+!Linear
+!Complex
+{{aligntop}}
+| Rowspan="2" |Tight
+|Examples
+|Dams, power grids, rail transport, marine transport
+|Nuclear power plants, DNA, chemical plants, aircraft, space missions, BANKS
+{{aligntop}}
+|Control method
+|'''Centralisation''': Best to deal with chain reactions, and best to deal with visible, expected linear reactions
+|'''Centralisation''': best to deal with chain reactions once they happen:
+Local control: best to deal with non-linear reactions and unexpected events as they happen.
+{{aligntop}}
+|Rowspan="2" |Loose
+|Examples
+|Manufacturing, single-purpose agencies
+|Mining, Research and development, multi-purpose agencies, universities
+{{aligntop}}
+|Control method
+|'''Centralisation or local control''': Few complex interactions; component failures create predictable results, and can be managed centrally.
+|'''Local control''': allows indigenous solutions where there is little risk of unstoppable chain reactions, and is best to deal with non-linear reactions and unexpected events as they happen
+|}
+And nor is greed — perhaps the thread that connects the capitalist entrepreneur to the socialist autocrat (let’s face it: it connects ''everyone'') any more causative — or, if it is, it is baked in to the human soul, so can’t really be solved for.
+Perrow thought it better to look at the by-product of these three modes as the problem in itself: ''[[Externality|externalities]]'': the social costs of the activity that are not reflected in its price, and borne by those who do not benefit from the activity. When the externality is powered by a tightly-coupled, non-linear system it can be out of all proportion to the bounties conferred on beneficiaries of that system — who are often a different class of individuals altogether. The Union Carbide accident at Bhopal being a great example: few of the half-million casualties would have bought a Duracell battery, let alone Union Carbide shareholders, and only 1,000 were employees.
+This led Perrow to frame his approach to the problem by reference to “catastrophic potential”, which may present itself as ''inherent'' catastrophic potential: the nature of the activity is tightly-coupled and non-linear such that no amount of reorganisation can prevent occasional system accidents, or  ''actual'' catastrophic potential: preventable shortcomings in any of the design, equipment, procedures, operators, supplies and materials, and environment, or component failures in the system could have catastrophic potential — these being things one can theoretically defend against, whereas inherent catastrophic potential is not; against in each case the cost of alternative solutions to the same problem.
+This leads to three categories of system. Those one should tolerate but seek to improve (mining, chemicals, dams, airways) those one should restrict (marine transport and DNA), and those one should abandon altogether, the benefits, however great, being out of all proportion with their downside risk. Here he includes nuclear weapons — no surprise — but also nuclear power).
 This is a long review already, so I should stop here. This is a fantastic book. It is somewhat hard to get hold of — there’s no audio version alas —but it is well worth the effort of trying.