The Unaccountability Machine

The Jolly Contrarian’s book review service™ →Go uptown The Unaccountability Machine: Why Big Systems Make Terrible Decisions - and How The World Lost Its Mind — Dan Davies, 2024. Index: Click ᐅ to expand: Book reviews Tell me more Sign up for our newsletter — or just get in touch: for ½ a weekly 🍺 you get to consult JC. Ask about it here.

’Tis neither malice, spite, nor virtue
Whose ledger swells, or plucks, the seedy fruits of progress —
But mainly accident.
Lest thee with surety know aught else —
Withhold thy assignations.

—Otto Büchstein, Die Schweizer Heulsuse

This review is sent from an unmonitored account. Please do not reply.

—Anon.

On a lack of barking dogs

The epic judicial process for inhouse legal in 2024 has been the Post Office Horizon IT scandal. It is not alone: Tom Hayes continues to appeal his conviction, and public disquiet continues to brew against Lucy Letby’s conviction, even if the criminal justice system is determined to ignore it.

In each case, colossal mortal engines — compliance departments, prudential regulators, health systems, audit systems, prosecuting agencies and the criminal justice system — were fully engaged to stop the kinds of behaviour alleged happening. In each case, the best we can say is that they failed. In most cases, it is a good deal worse than that: they prosecuted behaviour that did not happen.

Much modern business management exists specifically to prevent bad apples — or stupid apples — subverting the clever contrivances of thecomplicated systems we build. Its record at doing this is not good. You could (and JC does) catalogue a singular failure to achieve that basic end of “apple management”. LIBOR rigging and the sub-postmasters débâcle are but recent examples of a venerable tradition. Credit Suisse. FTX. Silicon Valley Bank. 1MDB. Archegos. The Fyre Festival. Enron. Woodstock 1999.

With all that infrastructure, superstructure and supervision how were a band of relatively lowly trading staff able to run riot? Did they run riot?

With all its infrastructure, internal and external legal advice, consultancy and (er) second sight, how did no one stop to think something must be wildly, catastrophically, wrong with the Post Office’s basic theory of the Horizon situation? How did no one, even once, apply Hanlon’s razor?

How could a popular team member, while maintaining a lively social life and without any obvious explanation, transform from a conscientious nurse into a calculating and flawless serial murderer?

Where were all the barking dogs?

Rogue apples, middle England and the grace of God

Either these are peculiar, localised problems — rogue gangs of bad apples plaguing the innocent houses of commerce and government — or the prevailing paradigm is in crisis and we need a new theory of the game. “Bad apples” are always the preferred diagnosis of mediocre management, after all. They relieve earnest executives of responsibility, leaving at most a deniable residue of blame for hiring the bad apples in the first place.

The LIBOR submitters fit the “rogues gallery” identikit nicely: by making microscopic adjustments inside an arcane process to which few were paying attention, they (allegedly) enriched themselves to the tune of millions of pounds while no one noticed. It was almost a victimless crime.

The Post Office’s bureaucratic middle managers do not fit that pattern. Few stood to gain from vilifying innocent sub-postmasters, and the potential rewards for those who did pale in comparison to city bonuses. These people did not seem psychopathic — unreflective, sure; narcissistic, in a few cases; illustrations of the Dunning-Kruger effect, all — but beyond observing the Buttocractic oath, their motivations were not base. They do not resemble “bad apples”. Their offence — and it is not a crime, though other commentators suggest it may have risen to that level — was weakness and credulity. A lack of spine.

These people are unremarkable, familiar, mediocre middle managers. Over-promoted; over-sure of their own competence; too mindful of nearby censorious trees to appreciate the calumny going on in the wood. In the halogen glare of cross-examined hindsight, their ineffectual interventions in an epic miscarriage of justice over an extended period were somewhere between regrettable and outrageous, but none of them resonate as odd. These people were adept at the sort of pencil-pushing, risk-averse buttocracy drilled by bitter experience into every in-house lawyer in the land. This is what in-house counsel do. This is how we behave.

We should ask ourselves: knowing what he knew then, would we have done anything differently? However tempting it is to take the moral high-ground from the safety of hindsight, we should not kid ourselves here.

Modern corporation as an unaccountability machine

Which brings us, finally, to Dan Davies’ new book. There is, he reports, a crisis of accountability in modern life: the relationship between “we” the general public and “we” the representatives and managers of the corporations who intermediate public life — many are on both sides of this equation, of course — has broken down. This is because modern corporations are designed to diffuse individual accountability for the actions a corporate legal entity takes, using what Davies calls “accountability sinks”. This is Davies’s example:

Someone — an airline gate attendant, for example — tells you some bad news; perhaps you’ve been bumped from the flight in favour of someone with more frequent flyer points. You start to complain and point out how much you paid for your ticket, but you’re brought up short by the undeniable fact that the gate attendant can’t do anything about it. You ask to speak to someone who can do something about it, but you’re told that’s not company policy.

The unsettling thing about this conversation is that you progressively realise that the human being you are speaking to is only allowed to follow a set of processes and rules that pass on decisions made at a higher level of the corporate hierarchy. It’s often a frustrating experience; you want to get angry, but you can’t really blame the person you’re talking to. Somehow, the airline has constructed a state of affairs where it can speak to you with the anonymous voice of an amorphous corporation, but you have to talk back to it as if it were a person like yourself.

An accountability sink arises when we delegate authority over a human process to a rulebook without giving anyone effective power to override it. Airlines, banks, ticketing agencies and online retailers give the public no access to anyone anymore: to confirm our order, or should it change, we are peremptorily notified by email from an unmonitored account. Because query handling takes time and costs money for no gain, the process is designed to impede the instinct to ask questions. By design, we can’t talk back.

The crushing organisational stasis in financial services — and plainly the Post Office — is a variety of the same thing. Even internally, policy, process and the imperative of scale discourage — at the limit, punish — independent thinking.

This design principle explains exactly why no-one at the Post Office saw, or appreciated the significance of or stopped to consider the implications of, the potentially incendiary advice they were receiving. It was not their job to second guess a process that had been set on rails well before they were involved.

In-house lawyers are the sweepers to the curling stones of a firm’s policies and business initiatives. Once a stone is launched, their job is to facilitate its progress, feverishly working away at the ice in front of it to preserve the momentum it already has. It is no part of their role to impede its progress.

In organisations over a certain size there is a presumption, not lightly rebutted, that others know what they are doing. This is its own accountability sink. When a decision appears to have been made elsewhere, whether by application of policy or exercise of someone’s discretion — questioning that decision is to sell a personal put option whose benefit, if there is one, accrues to the organisation, but whose loss allocates solely to you.

If you turn out to be right, someone else carries the can. If you’re not, you do. Either way, you have lost a friend.

True, you may head off an exceedingly remote tail risk if some day, years from now, the entire shabby affair is exposed and all participants held to public account before the nation’s watching eyes, but even then you will be lucky to end up at even money.

In-house counsel is not a moral compass

There is an argument, unstated in much commentary on the case, that the primary role of in-house counsel — of not just the GC when preparing briefings to the board, but all lawyers in the organisation — is to act as the organisation’s “moral compass”. Inhouse counsel, even more than their compliance colleagues, can sit above the fray, from where they can interrogate the organisation’s baser commercial instincts, at least insofar as they manifest in legal work product.

That’s a plausible theory, but it hardly reflects practice. For one thing, in-house legal is not part of the “operational stack”. It doesn’t see the great pitch and yaw of BAU activity that animates the firm’s mortal sinews. You wouldn’t situate your moral compass behind the fire extinguishers in the drawer with old chequebooks, broken torches and dead batteries if it were that important.

JC has a tongue in cheek history of in-house legal which charts the growth of the in-house legal function from a grey fellow in a cardigan who wordlessly managed the firm’s powers of attorney from a desk by the photocopiers to the weaponised 1,000 strong battle unit we know today.

No part of that transformation grew out of the yen for a stronger corporate conscience. It was about business facilitation, cost management (legal eagles were meant to be able to call bullshit on external advisors, not management) and the commodification of legal services. Precisely, the urge to codify processes, set operating rules and take away any need for anecdotal reflection about what the firm was about to do or where it was headed. In-house legal is, in this way, an anti moral compass: an accountability sink machine.

Nor should legal, or any risk function for that matter, be the firm’s conscience: the firm’s core values should be imbued in every one of its personnel, in everything they do. An operating model that supposes legal to be a crack morality squad, fighting a multi-front war to quell incipient infamy amongst the front office is absurd.

Management reports treat emerging legal risks as a known known: to be centrally tracked along with other project metrics against time and budget. Dashboards are expected to show uniform green across the board.

But what meaningful profit-generating business is so benign that we should expect its indicators always to be green? Allocating huge quantities of financial capital is inherently risky. If it weren’t, you would not be paid so much to do it. So should not risk managers seek out the biggest risks and highlight them, rather than papering them over and persuading management there is nothing to see?

Wouldn’t Credit Suisse’s board have been better served had the name “Archegos” appeared every month in its board papers for the two years leading up to its collapse as a major risk and revenue object? Might not that have changed the outcome for Credit Suisse of that collapse? Might it, perhaps, of made the collapse itself less likely?

A confederacy of stupid apples

It is easy, and tempting, to put the Horizon débâcle down to an unusual confederacy of stupid apples — this suits our personal self-esteem because if that is right, such a thing should not happen to us.

We should not be so sure.

“It would be nice,” says a valued correspondent, “if inhouse counsel were free to have a working moral compass inside their heads to help take open and ethically sustainable courses of action.”

It would. Just financially sustainable would be nice. But — thanks to the unaccountability machine — it is hardly likely. No-one got ahead by calling management out for incompetence.

Bear in mind, too, that servants, including in-house lawyers, are paid partisans for the firm. Advocates. Litigation, in the common law world, is an adversarial process, not a fact-finding enquiry.

Yes, there are standards of disclosure, fair dealing and honesty required but, upon finding weakness in one’s witness, a litigant’s instinct is not instantly to concede the case, but to find a better witness. Make a different argument. Lawyers are syllogism machines, not ethics machines. The rules oblige parties to disclose material evidence, but not flaky witnesses. Procedural rules make it hard to call hostile witnesses because you cannot cross examine them.

We are at danger of imputing to these individuals knowledge we now have — courtesy of the ITV drama, and months of cross-examined evidence — that they plainly did not. Not that the Horizon system was flimsy, biddable or prone to errors; they knew that — but that there was no fraud.

It seems absurd. It would be absurd were it not the plain facts of the matter. These people were told, by people they trusted and had no reason to doubt, that the sub-postmasters were fraudsters. That was their operating theory of the game. The fact that the Horizon system was a dog was an irritating fact getting in the way of a nobler pursuit of justice. These people thought they were on the right side of history. The same is undoubtedly true of the prosecution witnesses for Ms Letby.

Theory dependency

“The belief that science proceeds from observation to theory is still so widely and so firmly held that my denial of it is often met with incredulity.”

—Karl Popper, Conjectures and Refutations: The Growth of Scientific Knowledge

Humans are narratising animals. Before we can process information, we must create a framework in which it sits. Observation is theory-dependent.

If your operating theory is that you are on the right side of history — if that is your Bayesian prior — then you will filter all facts which might, by the lights of a different theory, suggest otherwise, accordingly: you will regard the postmaster who pelts you with intemperate emails about your own incompetence as unhinged. You will think the 15 page typed screed alleging breath-taking fraud against the former NASDAQ chairman as the ramblings of a lunatic. You will regard a condolence card to parents on the unexpected death of their infant not as an act of compassion and a meaningful continuation of care after a patient’s death, but the mark of a stone-cold psychopath.

We are prone to our own confirmation biases.

The accountability crisis in modern institutions will not be solved by rooting out the last “bad apple” — we don’t know who the bad apples are and keep rooting out the wrong ones in any case — nor doubling down on oversight and detection systems that are plainly not working as they are. These structures are part of the problem, because they diffuse responsibility rather than concentrating it. But that is what they are designed to do. These accountability sinks are not bugs but features. Until executives are made practically, directly and immediately accountable for the systems they preside over — not through eventual shaming, after years, through some public inquiry into what went wrong, but actually accountable, there and then, as the problems manifest — these scandals will keep coming.

It is hard to know what to prescribe here. Should we set up an adversarial system inside the organisation to mimic the court of public opinion outside it — a “defence department” overtly charged with taking the side of the alleged miscreant? Maybe we should. In any case, building a corporate culture more tolerant of dissent is at least a place to start.

See also

• Post Office Horizon IT scandal
• Mark-up the last deal
• There but for the grace of God go I