Talk:The Bayesian

Bayesian reasoning
/beɪːzˈiːən ˈriːzᵊnɪŋ/ (n.)

A method of statistical inference that updates an event’s probability from a general baseline probability as more information about the specific event becomes available .

Hot in the City

Like so many before them, Mike Lynch and Richard Gaunt, a pair of post-doctoral researchers into neural networks at Cambridge University, dreamed of commuting their intellectual achievements into city success. They had scaled the dreaming spires. They could see how revolutionary their technology would be: they were uniquely placed to lead the vanguard.

So, they founded a start up in 1996. They called it Autonomy. They imbued it with an aggressive, sales-led culture. They treated their top producers “like rock stars” and, allegedly, fired the bottom 5% of the sales force each quarter.

Vital Idol

Autonomy’s flagship product was IDOL: an “intelligent data operating layer” platform using, for the time, advanced pattern recognition and algorithmic inferential reasoning techniques to extract data from the morass of unstructured information with which modern corporations are inundated: documents, correspondence, contracts, instant messages, email and phone data.

IDOL promised a revolution: pure crystal business intelligence from an unnavigable swamp.

Now, socially awkward brainboxes hawking monstrous machines powered by mystical tools to convert oceans of muck into nuggets of brass had quite the cachet in 2005. It has quite the cachet now, come to think of it.

Autonomy targeted legal departments. One obvious application was in document management — a subject dear to the heart of any budding legal operationalist — so, unusually, we legal eagles were a prime target for the marketing blitz.

I remember it well: in a nutshell: awesome client entertainment; cool pitch; premium pricing; cluttered website, poor demo. Didn’t buy.

Got tickets to White Hart Lane, though and, a bit later, a catbird seat as a hurricane-force shitstorm blew up out of nowhere.

There are so many ironies in this story. Anyway.

White Wedding

The other city players noticed. Big Tech was not about to let the plucky Fenland brainboxes eat the world by themselves. They wanted in. In 2011, American megasaur Hewlett Packard pounced. Swooned by Lynch’s honeyed projections of hoverboards, time-travelling deLoreans and machines of loving grace watching over us and effortlessly converting the dreary white noise of the corporate monologue into monetisable intelligence, HP acquired Autonomy lock, stock and barrel US$11 billion.

This was 70% more than its prevailing market valuation. Autonomy’s board wasted no time in recommending the offer to shareholders.

The acquisition was an utter disaster: within the year HP ejected Lynch and wrote down its investment by, well, about 70%. Five years after that, it had flogged the whole thing off for glue.

The question arose: whose fault was that? On one hand, Autonomy’s presentations and projections were on the “optimistic” side. On the other, HP really wasn’t good at transformative acquisition: its previous deals with Compaq (2002), EDS (2008), and Palm (2010) had all been catastrophic. There was a pattern here.

(Forgot) to be a lover

Cue all manner of litigation. HP shareholders sued the HP board for its conduct of the acquisition — HP’s M&A strategy was notoriously bad. The SFO, SEC and FBI launched criminal investigations into potential fraudulent misrepresentation on Autonomy’s part. HP sued Lynch and his management team, accusing them of tricking HP into overpaying for the company. Lynch countered that HP had misunderstood the product, integrated it poorly into the HP business and then mismanaged it after acquisition. He might have also said “that, my dudes, is what the due diligence process is for”.

The various civil actions rumbled on for years.

In 2022, having spent USD100m pursuing its suit, HP won a pyrrhic billion dollars of the five it sought from Lynch. Lynch appealed.

Meanwhile, the FBI brought wire fraud and conspiracy charges against Lynch and his former Vice President of Finance, Stephen Chamberlain. The criminal standard of proof being much higher things went better for the Autonomy executives this time. In June 2024, Lynch and Chamberlain were acquitted on all counts.

Chamberlain returned to his home in Cambridgeshire. Lynch headed to the Mediterranean with his family, where he hosted some close friends on his superyacht, The Bayesian.

One night, one chance

On 17 August 2024, while jogging near his home, Stephen Chamberlain tragically was hit by a car. He died of his injuries two days later, on the 19th of August.

Early on the morning of that same day, the Bayesian was hit by a freak storm while at anchor off the coast of Sicily. She capsized and sank within 16 minutes, killing Lynch, his daughter and five others.

You could not, many were inclined to believe, make it up. The improbable circumstances of these accidents, within days of each other and just weeks after their acquittals, seemed an extraordinary coincidence, although the mainstream media quickly rationalised that an actual conspiracy here was implausible.

There were no obvious conspirators, for one thing HP had little to gain, and orchestrating any weather at all, let alone a freak storm powerful enough to sink a 550-ton yacht was surely beyond the coordinating powers of an organisation which has publicly struggled to manage basic due diligence.

Plus, if you did want to “off” an executive, there are easier ways than summoning up a biblical storm. Apparently.

Yet, still, the unfiltered maw of uninformed public speculation — from which I write, dear correspondent — found this all very, well, fishy.

That must have been some kind of storm. How often do massive ships sink, at anchor, in bad weather? Does that ever happen? What, in other words, are the odds of that?

And so we find ourselves looping reflexively into another one of an improbably large number of ironies — oop: there’s another one — with which this this story is shot through:

What the odds were is a question of Bayesian inference.

The Right Way

We hear more and more about “Bayesian reasoning”. This is a method of inferential reasoning developed by Thomas Bayes, a Presbyterian minister from Tunbridge Wells, who died in 1761. The theorem was found in his papers after his death, and published posthumously by his friend Richard Price.

Bayes’ theorem calculates the probability of this event being true (a “posterior probability”) based on the abstract general probability of events like this one in general being true (the “prior probability”) once you update it with new information about this specific event. Bayes devised a mathematical formula to calculate how these two pieces of information interact to give an updated posterior probability.

From a full deck, there is a ¼ (i.e., a 13/52) chance I will draw a heart. If I draw a second card, the odds change: are now only 51 cards. If I know the first card drawn was a heart, I can update the probability assessment: there is now a 12/51 chance it is a heart, and there are 13/51 chances for each of spades, diamonds and clubs. If I don't know see the first card, I have no new information so my assessment of the probabilities stays the same

This kind of reasoning dominates modern information technology. Automony’s IDOL platform was a pioneer in using Bayesian inferential techniques to analyse information. It was important enough to Mike Lynch to name his superyacht after it.

Still, prior probabilities are not magic. They can still mislead. Drawing a second, or a third consecutive heart is less likely but that is not to say it could not happen.

Bayesian inference might have told Mike Lynch, as he dropped anchor at Porticello, that his yacht was most unlikely to sink in the night.

Daytime drama

Here is a card game. The deck has just three cards: two clubs and a heart. The object of the game is to end up with the heart.

The dealer shuffles and deals the three cards face down on the table, in the “river”. The player chooses one card from the river, but it stays face down. The dealer turns one of the remaining cards in the river, to reveal a club.

The player now has the choice to stay with her original card, or switch it for the remaining face-down card in the river.

What should the player do? Stick, or twist?

There is no better example of Bayesian inference at work than the Monty Hall problem. What follows will seem either trite, or completely Earth-shattering, depending on whether you have come across it before or not. It indicates, too, how tremendously bad our instinctive grasp of probabilities is:

You are a game-show contestant. The host shows you three doors and tells you: “Behind one of those doors is a Ferrari. Behind the other two are goats. You may choose one door.

Knowing you have a ⅓ chance, you choose a door at random.

Now the host theatrically opens one of the doors you didn’t choose, revealing a goat.

Two closed doors remain. She offers you the chance to reconsider your choice.

Do you stick with your original choice, switch, or does it not make a difference?

Intuition suggests it makes no difference. At the beginning, each door carries an equal probability: ⅓, After the reveal, the remaining doors still do: ½.

So, while your odds have improved, the odds remain equal for each unopened door. So, it still doesn’t matter which you choose: Right?

Wrong. The best odds are if you switch: there remains a ⅓ chance the car is behind the first door you picked; there is now a ⅔ chance the Ferrari is behind the other door. Staying put is to commit to a choice you made then the odds were worse.

We know this thanks to Bayesian inference. There are two categories of door; ones you chose, and ones you didn’t. There’s only one door in the “chosen” category and two doors in the “unchosen” category. At the start you knew each was equally likely to hold the car. This was the “prior probability”. There was a ⅓ chance per door or, if we categorise the doors, a ⅓ chance it was behind a chosen door and a ⅔ chance it was behind an unchosen door.

Then you got some updated information, but only about the “unchosen door” category: One of those doors definitely doesn’t hold the car. You have no new information about the “chosen door” category, however.

You can update your prior probability estimates about the unchosen doors. One now has a zero chance of holding the car. Therefore, it follows the other door has a ⅔ chance. All the odds of the unchosen category now sit behind its single unopened door.

Therefore you have a better chance of winning the car (though not a certainty — one time in three you’ll lose) if you switch.

Bayesian inference invites us to update our assessment of probabilities based on what we have since learned. We do this quite naturally in certain environments — when playing cards, for example: once the Ace, has been played, you know your King is high, and that to update a Bayesian prior— but not in others. We tend to be better at Bayesian reasoning in familiar circumstances, like card games and not in novel ones like Ferrari and goat competitions.

Come on, come on

We are pattern-seeking animals. Narratisers. We are drawn to dramatic stories with heroes and villains. When key figures in a meme-cluster of ill-advised litigation tragically die in freak conditions we demand an explanation. It does not seem satisfactory to conclude it was “just one of those things”.

We are tempted to update the probability assessment dramatically because of the timing and the freak nature of the storm in Sicily. But here Bayesian inference — the mathematical foundations for Autonomy’s IDOL machines — tell us a different story: sometimes patterns are just noise. Ships sink — especially ships with unusually tall masts and retractable keels. Sometimes joggers get hit by cars. Not often, but more often than corporations summon freak storms and coordinate random traffic to commit executive murder.

The sceptical principle “absence of evidence is not evidence of absence” holds only where we have not looked for evidence. If you have, and there is none, there is evidence of absence. Sometimes there aren’t enough data to update our “priors”. The rational response is to acknowledge the limitations of our inference — even when the coincidences seem to cry out for a more dramatic explanation.

HP’s folly was to draw too many conclusions from too little data during its Autonomy due diligence. We should avoid making the same mistake when analysing tragic events.