The Jolly Contrarian

Normal Accidents: Living with High-Risk Technologies: Difference between revisions

Page Discussion

Normal Accidents: Living with High-Risk Technologies (view source)

Revision as of 09:30, 2 October 2020

486 bytes added ,  2 October 2020
no edit summary
No edit summary
No edit summary
Line 22: Line 22:
Where you have a complex system, you should therefore ''expect'' accidents — yes, and opportunities, quirks and serendipities, to be sure, but here we are talking about risk — to arise from unexpected, [[non-linear interaction]]s. Such accidents, says Perrow, are “normal”, not in the sense of being regular or expected, but in the sense that ''it is an inherent property of the system to have this kind of accident at some point or other''.<ref>In the forty-year operating history of nuclear power stations, there had (at the time of writing!) been ''no'' catastrophic meltdowns, “... but this constitutes only an “industrial infancy” for complicated, poorly understood transformation systems.” Perrow had a chilling prediction: “... the ingredients for such accidents are there, and unless we are very lucky, one or more will appear in the next decade and breach containment.” Ouch.</ref>
Where you have a complex system, you should therefore ''expect'' accidents — yes, and opportunities, quirks and serendipities, to be sure, but here we are talking about risk — to arise from unexpected, [[non-linear interaction]]s. Such accidents, says Perrow, are “normal”, not in the sense of being regular or expected, but in the sense that ''it is an inherent property of the system to have this kind of accident at some point or other''.<ref>In the forty-year operating history of nuclear power stations, there had (at the time of writing!) been ''no'' catastrophic meltdowns, “... but this constitutes only an “industrial infancy” for complicated, poorly understood transformation systems.” Perrow had a chilling prediction: “... the ingredients for such accidents are there, and unless we are very lucky, one or more will appear in the next decade and breach containment.” Ouch.</ref>


Are financial systems [[complex]]? About as complex as any distributed system known to humankind. Are they tightly coupled? Well, you could ask the principals of [[LTCM]], [[Enron]], Bear Stearns, Amaranth Advisors, [[Lehman]] brothers or Northern Rock, if any of those venerable institutions were still around to tell you about it. But yes. Might mortgage securitisations have been on Perrow’s mind?
Are financial systems [[complex]]? About as complex as any distributed system known to humankind. Are they tightly coupled? Well, you could ask the principals of [[LTCM]], [[Enron]], Bear Stearns, Amaranth Advisors, [[Lehman]] brothers or Northern Rock, if any of those venerable institutions were still around to tell you about it. But yes. Might reckless mortgage securitisation, excess [[leverage]] and flash boys have been on Perrow’s mind? We rather think so: “New financial instruments such as [[Financial weapons of mass destruction|derivatives]] and [[hedge fund]]s and new techniques such as [[high-frequency trading|programmed trading]] further increase the complexity of interactions. ''Breaking up a loan on a home into tiny packages and selling them on a world-wide basis increases interdependency.''”<ref>{{br|Normal Accidents}} p. 385.</ref> He wrote this in 1999, for Pete’s sake.
:''New financial instruments such as derivatives and hedge funds and new techniques such as programmed trading further increase the complexity of interactions. Breaking up a loan on a home into tiny packages and selling them on a world-wide basis increases interdependency.''<ref>{{br|Normal Accidents}} p. 385. This in 1999, for Pete’s sake</ref>


===How to deal with [[system accidents]]===
===How to deal with [[system accidents]]===
Line 45: Line 44:


===“Operator error” is almost always the wrong answer===
===“Operator error” is almost always the wrong answer===
Human beings being system components, it is rash to blame for failure a component constitutionally disposed to fail, even when not put in a position, through system design or economic incentive — a ship’s captain being expected to work a 48-hour watch — where failure is more or less inevitable (Perrow calls these “forced operator errors”).
Human beings being system components, it is rash to blame them when they are component that is constitutionally disposed to fail — we are frail, mortal, inconstant, narratising beings — even when not put in a position, through system design or economic incentive that makes failure inevitable. A ship’s captain who is expected to work a 48-hour watch and meet unrealistic deadlines is hardly positioned, let alone incentivised to prioritise safety. Perrow calls these “forced operator errors”.
 
:''But again, “operator error” is an easy classification to make. What really is at stake is an inherently dangerous working situation where production must keep moving and risk-taking is the price of continued employment.<ref>{{br|Normal Accidents}} p. 249.</ref>  
:''But again, “operator error” is an easy classification to make. What really is at stake is an inherently dangerous working situation where production must keep moving and risk-taking is the price of continued employment.<ref>{{br|Normal Accidents}} p. 249.</ref>  
If an operator's role is simply to carry out a tricky but routine part of the system then the inevitable march of technology makes this ever more fault of design and not personnel: humans, we know, are not good computers. They are good at figuring out what to do when something unexpected happens; making decisions; exercising judgment. But they — ''we'' — are ''lousy'' at doing repetitive tasks and following instructions. As ''The Six Million Dollar Man'' had it, ''we have the technology''. We should damn well use it.
 
If, on the other hand, the operator’s role is to manage ''complexity'' — then technology, checklists and pre-packaged risk taxonomies will be of little use. Perrow’s account of the control deck at Three Mile Island is instructive:
If an operator’s role is simply to carry out a tricky but routine part of the system then the inevitable march of technology makes this ever more a fault of design and not personnel: humans, we know, are not good computers. They are good at figuring out what to do when something unexpected happens; making decisions; exercising judgment. But they — ''we'' — are ''lousy'' at doing repetitive tasks and following instructions. As ''The Six Million Dollar Man'' had it, ''we have the technology''. We should damn well use it.
 
If, on the other hand, the operator’s role is to manage ''[[complexity]]'' — then technology, checklists and pre-packaged risk taxonomies can only take you so far and, at the limit, can get in the way. Perrow’s account of the control deck at Three Mile Island is instructive:
 
:''Besides, about this time—just four or five minutes into the accident—another more pressing problem arose. The reactor coolant pumps that had turned on started thumping and shaking. They could be heard and felt from far away in the control room. Would they withstand the violence they were exposed to? Or should they be shut off? A hasty conference was called, and they were shut off. (It could have been, perhaps should have been, a sign that there were further dangers ahead, since they were “cavitating”—not getting enough emergency coolant going through them to function properly.) In the control room there were three audible alarms sounding, and many of the 1,600 lights (on-off lights and rectangular displays with some code numbers and letters on them) were on or blinking. The operators did not turn off the main audible alarm because it would cancel some of the annunciator lights. The computer was beginning to run far behind schedule; in fact it took some hours before its message that something might be wrong with the PORV finally got its chance to be printed. Radiation alarms were coming on. The control room was filling with experts; later in the day there were about forty people there. The phones were ringing constantly, demanding information the operators did not have. Two hours and twenty minutes after the start of the accident, a new shift came on. <ref>{{br|Normal Accidents}} p. 28.</ref>  
:''Besides, about this time—just four or five minutes into the accident—another more pressing problem arose. The reactor coolant pumps that had turned on started thumping and shaking. They could be heard and felt from far away in the control room. Would they withstand the violence they were exposed to? Or should they be shut off? A hasty conference was called, and they were shut off. (It could have been, perhaps should have been, a sign that there were further dangers ahead, since they were “cavitating”—not getting enough emergency coolant going through them to function properly.) In the control room there were three audible alarms sounding, and many of the 1,600 lights (on-off lights and rectangular displays with some code numbers and letters on them) were on or blinking. The operators did not turn off the main audible alarm because it would cancel some of the annunciator lights. The computer was beginning to run far behind schedule; in fact it took some hours before its message that something might be wrong with the PORV finally got its chance to be printed. Radiation alarms were coming on. The control room was filling with experts; later in the day there were about forty people there. The phones were ringing constantly, demanding information the operators did not have. Two hours and twenty minutes after the start of the accident, a new shift came on. <ref>{{br|Normal Accidents}} p. 28.</ref>  


This is, as Perrow sees it, the central dilemma of the complex system. The nature of normal accidents is such that they need experienced, wise operators on the ground ready to think quickly and laterally to solve unfolding problems, but the enormity of the risks involved mean that  
This is, as Perrow sees it, the central dilemma of the [[complex system]]. The nature of [[normal accidents]] is such that they need experienced, wise operators on the ground ready to think quickly and laterally to solve unfolding problems, but the enormity of the risks involved mean that central management are not prepared to delegate so much responsibility to the mortal, inconstant, narratising [[meatware]].
 


{{sa}}
{{sa}}
*[[Complexity]]
*[[Complexity]]
{{ref}}
{{ref}}
Retrieved from "https://jollycontrarian.com/index.php?title=Special:MobileDiff/47502"