Talk:The Bayesian: Difference between revisions
Amwelladmin (talk | contribs) |
Amwelladmin (talk | contribs) Tags: Mobile edit Mobile web edit Advanced mobile edit |
||
| (9 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=== The acquisition and trial === | === The acquisition and trial === | ||
Autonomy was a British tech darling of the early 2000s founded by Cambridge mathematics PhD Mike Lynch. Autonomy's flagship product was IDOL: Intelligent data operating layer a platform using at the time advanced pattern recognition techniques to extract management information and data from the sort of unstructured information with which most corporations are inundated — documents, correspondence, contracts, instant messages, email and phone data. | |||
Hewlett Packard acquired Autonomy in 2011 for US$11 billion. This was a 70% premium over Autonomy’s stock market valuation. | |||
It may not, therefore, be a surprise to the reader — though it was to HP’s executive — that the acquisition was a disaster: HP was unable to make the business workable and eventually wrote down its investment by $8.8 billion. | |||
HP sued Lynch and his management team, accusing them of fraudulent accounting, overstating earnings and revenue, and misleading HP into overpaying for the company. The litigation went on for years. | |||
In 2022, having spent an estimated USD100m prosecuting it, HP won its case and Lynch was ordered to pay US$1bn in damages (of the US$5bn HP had claimed). | |||
Now, as [[Matt Levine]] often reminds us, in the US, everything is securities fraud. While the civil case continued, Lynch and Autonomy’s former Vice President of Finance Stephen Chamberlain were charged with criminal wire fraud and conspiracy and in May 2023, extradited to the US to face criminal trial. | |||
In June 2024, Lynch and Chamberlain were acquitted on all counts. The jury was not persuaded [[beyond reasonable doubt]] that Lynch had intentionally committed fraud. | |||
After the acquittal, Chamberlain returned to his home in Cambridgeshire while, by way of celebration, Lynch treated his daughter and some close friends to a fortnight on his superyacht, ''The Bayesian'' in the Mediterranean. | |||
On 17 August 2024 Stephen Chamberlain was hit by a car while jogging in Longstanton Cambridgeshire. He died of his injuries two days later. | |||
Early on the morning of the same day, while at anchor half a mile off the port of Porticello on the north coast of Sicily, ''The Bayesian'' was hit by a freak storm and, in the space of about 16 minutes, capsized and sank, tragically killing Lynch, his daughter and five others. | |||
===Conspiracy theory=== | ===Conspiracy theory=== | ||
The improbable circumstances of | The improbable circumstances of these accidents, within two days of each other and just weeks after their acquittals, raised eyebrows. It seemed, if nothing else, to be an ''extraordinary'' coincidence although the mainstream media quickly rationalised that an actual conspiracy here was highly unlikely — no one had anything obvious to gain, for one thing, and orchestrating any freak storm at all, let alone powerful enough to capsize and sink a 55-metre, 550-ton yacht is beyond the capacity even of the deep state. If you wanted to “off” a business executive, there were ''far'' easier ways of doing it. | ||
Yes, still, the unfiltered maw of uninformed public speculation — from which I write, dear correspondent — found this all very fishy. | |||
How could a $40m state-of-the-art superyacht, crewed by experienced mariners just ''sink'', in sixteen minutes, while at anchor, during a summer Mediterranean storm? | |||
===Conspiracy v irony: Bayesian reasoning=== | |||
We hear more and more about “[[Bayesian reasoning]]”. This method of statistical reasoning derived from the work of Thomas Bayes, an (at the time) obscure 18th-century Presbyterian minister from Tunbridge Wells, now dominates modern information technology and was important enough to Mike Lynch's business for him to name a £30m yacht in its honour. | |||
By contrast, systems theory says in a nutshell, into a bit more complicated than that. In the case of great artists and great visionary business people, their input into the artistic process is not discounted altogether but instead aggregated with a great deal of other system information to generate an outcome. Shakespeare was indeed a genius but would yet have died in anonymity were it not for his sponsors, publishers, patrons, theatres, actors, critics and audience: the | {{qd|Bayesian reasoning|/beɪːzˈiːən ˈriːzᵊnɪŋ/|n|A method of statistical inference that updates a probability as more information becomes available about the hypothesis. Bayesian reasoning ensures comprehensive use of available information: | ||
{{L1}}improves risk assessment and forecasting.<li> | |||
is used in artificial intelligence to allow machine learning algorithms to learn and adapt from their data more effectively. <li> | |||
helps evaluate evidence and enables more informed judgments, ensuring that decisions are based on a comprehensive analysis of all available information</ol>}} | |||
Automony’s Intelligent Data Operating Layer (IDOL) system used Bayesian inference to manage and analyze unstructured data, enabling it to make probabilistic predictions and improve information retrieval. | |||
===Blame and exculpation=== | |||
{{drop|T|he blame and}} recrimination process started quickly. Italian police launch an investigation into the incident raising the possibility of manslaughter or culpable shipwreck charges against the Bayesian’s skipper, chief engineer and the sailor on watch duty at the time of the incident.<ref>https://apnews.com/article/italy-sicily-superyacht-sinking-investigation-7b26e40e9efd69c4d08e189093f67ad9</ref> | |||
In the meantime, the Bayesian’s boatbuilder The Italian Sea Group launched, then quickly disowned, a civil suit against the Bayesian’s crew and owner — a company controlled by Mike Lynch’s widow Angla Bacares — seeking compensation for reputational damage and loss of earnings, alleging among other things that the crew was inappropriately selected, did not make necessary preparations for the storm despite advanced weather warnings, and their actions during the storm contributed to the sinking. | |||
TISG itself is under police investigation in connection with the tragedy: this may explain its precipitate behaviour in launching formal legal proceedings: one way of getting ahead of suspicion as a wrongdoer is to cast yourself as the victim. | |||
===Linear and systems: two competing theories=== | |||
{{quote| A conspiracy theorist is someone who’s never tried to organise a surprise party. | |||
— John F. Kennedy}} | |||
Hold this as a provisional hypothesis: We can bifurcate explanations of the world into [[Linear theory|''linear'' theories]] and [[systems theory|''systems'' theories]]. | |||
“Linear theories” view the world as a linear [[complicated system]].<ref>[[Complicated system]]s are bounded interactive processes: they involve interaction with autonomous agents but within fixed boundaries and according to preconfigured, known and static rules of engagement. All relevant information is available to, even if not necessarily known by, all participants in the system.</ref> They maybe forward or backward engineered. It may require great skill to navigate a complicated system but they are nevertheless in theory predictable, in the sense that all events are caused and have effects. One can regard the behaviour of the system as a case of causes and effects. There are “root causes” for what happens which dominate the incidental circumstances in which they operate. The inspirational CEO, the star striker, the [[Bad apple|bad apples]] who ruined it for everyone, the [[operator]] whose human error caused the air crash: these are linear explanations. | |||
Conspiracy theories are typically linear explanations in that they put ultimate blame (or credit) for a given state of affairs things down to the intentional actions, be they malign or well-meant, of a limited number of disproportionately influential people. | |||
“Systems theories” view the world as a non-linear [[complex system]].<ref>[[Complex systems]] are “unbounded, interactive process. Involves interaction with autonomous agents without boundaries, without pre-agreed rules, and where information is limited and asymmetric. Rules, boundaries and each participant’s objectives are dynamic and change interactively. Impossible to predict.”</ref> attribute outcomes to the behaviour of a wider interlocking system of relationships where individuals’ acumen or motivations contribute to, but rarely determine the outcomes the system produces. They are usually non-linear: in a [[complex system]] ''unexpectable'' things can and do happen. This is generally not a single person’s fault, but an unexpected consequence of the design of the system. As systems experience unexpected consequences they tend to learn and adjust to them. | |||
Old systems, having been around for longer and having been exposed to more variations in condition, are better stress-tested and therefore throw up fewer unexpected consequences than ''new'' systems.<ref>This is sometimes called the “[[Lindy effect]]”, but is also explained in terms of [[Pace layering|pace layering]]: old systems occupy deeper layers.</ref> | |||
I will grant you at once that this is a wide conception indeed of “conspiracy theory”: it includes not just gunpowder plots and Russian bots in Western elections but the general idea that great art is the product of singular genius, commercial success is the outcome of exceptional leadership, and jazz is not just a succession of happy accidents. | |||
By contrast, systems theory says in a nutshell, into a bit more complicated than that. In the case of great artists and great visionary business people, their input into the artistic process is not discounted altogether but instead aggregated with a great deal of other system information to generate an outcome. Shakespeare was indeed a genius but would yet have died in anonymity were it not for his sponsors, publishers, patrons, theatres, actors, critics and audience: the magnificent cultural establishment that we now know as the Shakespeare canon contains a lot of stuff that was nothing to do with William Shakespeare. | |||
====Systems and paradigms==== | ====Systems and paradigms==== | ||
| Line 28: | Line 67: | ||
====Systemantics==== | ====Systemantics==== | ||
The best place to start [[systems theory]] is {{author|John Gall}}’s short, acerbic, funny and devastatingly incisive book {{br|Systemantics: The Systems Bible}}. System theories have an acronym: “POSIWID”: the “''purpose of a system is what it does''”. This, Gall gently points out, Is by inevitable outcome not what those who designed the system had in mind. The System tends to oppose its own intended function so therefore to blame conspirators Who occupy positions of ostensible influence and power within the system is rather to miss the point. They are as much victims of systemantics as anyone else. | The best place to start [[systems theory]] is {{author|John Gall}}’s short, acerbic, funny and devastatingly incisive book {{br|Systemantics: The Systems Bible}}. System theories have an acronym: “POSIWID”: the “''purpose of a system is what it does''”. This, Gall gently points out, Is by inevitable outcome not what those who designed the system had in mind. The System tends to oppose its own intended function so therefore to blame conspirators Who occupy positions of ostensible influence and power within the system is rather to miss the point. They are as much victims of systemantics as anyone else. | ||
====Operator error==== | |||
====Defamation==== | |||
The claim was brought at Italian law, and withdrawn before anyone got much of a look at it, but it is fun to imagine it was essentially one of [[defamation]]: | |||
{{Qd|Defamation|ˌdɛfəˈmeɪʃᵊn|n|A false statement about a person, communicated to others, that harms the person’s reputation. | |||
The statement must be untrue, specific enough to identify the person and published or shared with third parties such that it damages the person’s reputation or standing.}} | |||
Can your careless ''conduct'' — not “representational” gestures; actually non-communicative acts, like steering (or not steering) a boat — be a ''statement''? | |||
The defamatory “statement”, we presume, would be that ''the boat was designed so badly that, even at anchor, a crew of competent professionals sailors could not stop it scuttling in a severe storm''. | |||
That allegation would certainly be defamatory to a shipbuilder. ''If it were untrue''. | |||
But the problem the shipbuilder has is that the ship ''did'' sink at anchor in a storm. This does normally happen to well designed ships. Presuming it was well designed, then either (i) the storm’s ferocity or (ii) the sailors’ negligence, was truly unprecedented. | |||
Revision as of 20:06, 22 September 2024
The acquisition and trial
Autonomy was a British tech darling of the early 2000s founded by Cambridge mathematics PhD Mike Lynch. Autonomy's flagship product was IDOL: Intelligent data operating layer a platform using at the time advanced pattern recognition techniques to extract management information and data from the sort of unstructured information with which most corporations are inundated — documents, correspondence, contracts, instant messages, email and phone data.
Hewlett Packard acquired Autonomy in 2011 for US$11 billion. This was a 70% premium over Autonomy’s stock market valuation.
It may not, therefore, be a surprise to the reader — though it was to HP’s executive — that the acquisition was a disaster: HP was unable to make the business workable and eventually wrote down its investment by $8.8 billion.
HP sued Lynch and his management team, accusing them of fraudulent accounting, overstating earnings and revenue, and misleading HP into overpaying for the company. The litigation went on for years.
In 2022, having spent an estimated USD100m prosecuting it, HP won its case and Lynch was ordered to pay US$1bn in damages (of the US$5bn HP had claimed).
Now, as Matt Levine often reminds us, in the US, everything is securities fraud. While the civil case continued, Lynch and Autonomy’s former Vice President of Finance Stephen Chamberlain were charged with criminal wire fraud and conspiracy and in May 2023, extradited to the US to face criminal trial.
In June 2024, Lynch and Chamberlain were acquitted on all counts. The jury was not persuaded beyond reasonable doubt that Lynch had intentionally committed fraud.
After the acquittal, Chamberlain returned to his home in Cambridgeshire while, by way of celebration, Lynch treated his daughter and some close friends to a fortnight on his superyacht, The Bayesian in the Mediterranean.
On 17 August 2024 Stephen Chamberlain was hit by a car while jogging in Longstanton Cambridgeshire. He died of his injuries two days later.
Early on the morning of the same day, while at anchor half a mile off the port of Porticello on the north coast of Sicily, The Bayesian was hit by a freak storm and, in the space of about 16 minutes, capsized and sank, tragically killing Lynch, his daughter and five others.
Conspiracy theory
The improbable circumstances of these accidents, within two days of each other and just weeks after their acquittals, raised eyebrows. It seemed, if nothing else, to be an extraordinary coincidence although the mainstream media quickly rationalised that an actual conspiracy here was highly unlikely — no one had anything obvious to gain, for one thing, and orchestrating any freak storm at all, let alone powerful enough to capsize and sink a 55-metre, 550-ton yacht is beyond the capacity even of the deep state. If you wanted to “off” a business executive, there were far easier ways of doing it.
Yes, still, the unfiltered maw of uninformed public speculation — from which I write, dear correspondent — found this all very fishy.
How could a $40m state-of-the-art superyacht, crewed by experienced mariners just sink, in sixteen minutes, while at anchor, during a summer Mediterranean storm?
Conspiracy v irony: Bayesian reasoning
We hear more and more about “Bayesian reasoning”. This method of statistical reasoning derived from the work of Thomas Bayes, an (at the time) obscure 18th-century Presbyterian minister from Tunbridge Wells, now dominates modern information technology and was important enough to Mike Lynch's business for him to name a £30m yacht in its honour.
Bayesian reasoning
/beɪːzˈiːən ˈriːzᵊnɪŋ/ (n.)
A method of statistical inference that updates a probability as more information becomes available about the hypothesis. Bayesian reasoning ensures comprehensive use of available information:
- improves risk assessment and forecasting.
- is used in artificial intelligence to allow machine learning algorithms to learn and adapt from their data more effectively.
- helps evaluate evidence and enables more informed judgments, ensuring that decisions are based on a comprehensive analysis of all available information
Automony’s Intelligent Data Operating Layer (IDOL) system used Bayesian inference to manage and analyze unstructured data, enabling it to make probabilistic predictions and improve information retrieval.
Blame and exculpation
The blame and recrimination process started quickly. Italian police launch an investigation into the incident raising the possibility of manslaughter or culpable shipwreck charges against the Bayesian’s skipper, chief engineer and the sailor on watch duty at the time of the incident.[1]
In the meantime, the Bayesian’s boatbuilder The Italian Sea Group launched, then quickly disowned, a civil suit against the Bayesian’s crew and owner — a company controlled by Mike Lynch’s widow Angla Bacares — seeking compensation for reputational damage and loss of earnings, alleging among other things that the crew was inappropriately selected, did not make necessary preparations for the storm despite advanced weather warnings, and their actions during the storm contributed to the sinking.
TISG itself is under police investigation in connection with the tragedy: this may explain its precipitate behaviour in launching formal legal proceedings: one way of getting ahead of suspicion as a wrongdoer is to cast yourself as the victim.
Linear and systems: two competing theories
A conspiracy theorist is someone who’s never tried to organise a surprise party.
— John F. Kennedy
Hold this as a provisional hypothesis: We can bifurcate explanations of the world into linear theories and systems theories.
“Linear theories” view the world as a linear complicated system.[2] They maybe forward or backward engineered. It may require great skill to navigate a complicated system but they are nevertheless in theory predictable, in the sense that all events are caused and have effects. One can regard the behaviour of the system as a case of causes and effects. There are “root causes” for what happens which dominate the incidental circumstances in which they operate. The inspirational CEO, the star striker, the bad apples who ruined it for everyone, the operator whose human error caused the air crash: these are linear explanations.
Conspiracy theories are typically linear explanations in that they put ultimate blame (or credit) for a given state of affairs things down to the intentional actions, be they malign or well-meant, of a limited number of disproportionately influential people.
“Systems theories” view the world as a non-linear complex system.[3] attribute outcomes to the behaviour of a wider interlocking system of relationships where individuals’ acumen or motivations contribute to, but rarely determine the outcomes the system produces. They are usually non-linear: in a complex system unexpectable things can and do happen. This is generally not a single person’s fault, but an unexpected consequence of the design of the system. As systems experience unexpected consequences they tend to learn and adjust to them.
Old systems, having been around for longer and having been exposed to more variations in condition, are better stress-tested and therefore throw up fewer unexpected consequences than new systems.[4]
I will grant you at once that this is a wide conception indeed of “conspiracy theory”: it includes not just gunpowder plots and Russian bots in Western elections but the general idea that great art is the product of singular genius, commercial success is the outcome of exceptional leadership, and jazz is not just a succession of happy accidents.
By contrast, systems theory says in a nutshell, into a bit more complicated than that. In the case of great artists and great visionary business people, their input into the artistic process is not discounted altogether but instead aggregated with a great deal of other system information to generate an outcome. Shakespeare was indeed a genius but would yet have died in anonymity were it not for his sponsors, publishers, patrons, theatres, actors, critics and audience: the magnificent cultural establishment that we now know as the Shakespeare canon contains a lot of stuff that was nothing to do with William Shakespeare.
Systems and paradigms
I rabbit on a lot on this site about power structures and paradigms. These are systems of political, scientific and cultural control.
Systemantics
The best place to start systems theory is John Gall’s short, acerbic, funny and devastatingly incisive book Systemantics: The Systems Bible. System theories have an acronym: “POSIWID”: the “purpose of a system is what it does”. This, Gall gently points out, Is by inevitable outcome not what those who designed the system had in mind. The System tends to oppose its own intended function so therefore to blame conspirators Who occupy positions of ostensible influence and power within the system is rather to miss the point. They are as much victims of systemantics as anyone else.
Operator error
Defamation
The claim was brought at Italian law, and withdrawn before anyone got much of a look at it, but it is fun to imagine it was essentially one of defamation:
Defamation
ˌdɛfəˈmeɪʃᵊn (n.)
A false statement about a person, communicated to others, that harms the person’s reputation.
The statement must be untrue, specific enough to identify the person and published or shared with third parties such that it damages the person’s reputation or standing.
Can your careless conduct — not “representational” gestures; actually non-communicative acts, like steering (or not steering) a boat — be a statement?
The defamatory “statement”, we presume, would be that the boat was designed so badly that, even at anchor, a crew of competent professionals sailors could not stop it scuttling in a severe storm.
That allegation would certainly be defamatory to a shipbuilder. If it were untrue.
But the problem the shipbuilder has is that the ship did sink at anchor in a storm. This does normally happen to well designed ships. Presuming it was well designed, then either (i) the storm’s ferocity or (ii) the sailors’ negligence, was truly unprecedented.
- ↑ https://apnews.com/article/italy-sicily-superyacht-sinking-investigation-7b26e40e9efd69c4d08e189093f67ad9
- ↑ Complicated systems are bounded interactive processes: they involve interaction with autonomous agents but within fixed boundaries and according to preconfigured, known and static rules of engagement. All relevant information is available to, even if not necessarily known by, all participants in the system.
- ↑ Complex systems are “unbounded, interactive process. Involves interaction with autonomous agents without boundaries, without pre-agreed rules, and where information is limited and asymmetric. Rules, boundaries and each participant’s objectives are dynamic and change interactively. Impossible to predict.”
- ↑ This is sometimes called the “Lindy effect”, but is also explained in terms of pace layering: old systems occupy deeper layers.