Big data

The JC’s amateur guide to systems theory™ →Go uptown Index: Click ᐅ to expand: Systems theory The Devil’s Advocate Tell me more Sign up for our newsletter — or just get in touch: for ½ a weekly 🍺 you get to consult JC. Ask about it here.

Turkey: “I have transformed myself into a data-driven business. All my data — and I’ve got reams of the stuff — tells me that every morning I shall be fed at 9 am on the dot. Aha! Here comes the farmer, right on time! I wonder if I’ll get a special treat because it is Christmas!”

Charlotte (spinning web): Ummm

As at the time of its analysis, all data is from the past.

—Roger Martin

The obsession with big data has a few implications:

It is premium mediocre

Firstly it expresses a preference for the aggregate over the specific, and the average over the outlier, the individual, the unique or extraordinary. It is to prefer the mediocre, for its weight of numbers, over the isolated vision of a genius or the depravity of the ugliest man.

As surely as the ugliest man killed God, so did data kill the superman. The will to power is defeated by the million-strong dull blades of the will to entropy. It is the will to premium mediocre.

It is historical

All data is from the past, as the saying goes

It is illiberal

Second, in its reductionism, in its funnelling of a dispersed population into an essential homogeneity, it speaks to the underlying belief in a grand unifying theory of everything: a transcendent truth. This, in the JC’s view, is a profoundly illiberal idea: to be unable to accommodate pluralism is to deny of pluralism.

It is noisy

Thirdly, to embrace all the data you can find is to degrade the signal-to-noise ratio. Even if you buy into the incoherent reductionist idea that the “signal” is some kind of transcendent truth, by industrialising your data, you risk burying it and if you don’t — if like we pluralists you see any signal as not just a suitable narrative for your present purposes, the more data you gather, the more possible narratives — conflicting narratives; incommensurable narratives — you will have. Now this is, for a pluralist, is a good thing: every narrative is a tool in your workshop, the more you have the better you are equipped to deal with the unknown unknowns our complex world will surely throw at us — but that tends not to be what big data disciples are after.

It is not a universal affirmative

Even if, from pure data, you could establish the causal relationship between data you have observed and an event that drives it (it is axiomatic that you can’t, by the way: you can only derive a correlation, and we know how spurious those can be) you still can’t conclude that the cause propelling the general is the same one that compelled any particular.

Averages are crappy things to aspire to, or configure your business to, for a number of reasons .

Because the machinations of statistics can, in certain contexts, inflame the passions of the righteous, the JC has devised the parable of the squirrels to tease this out.

See also

• Signal-to-noise ratio
• In God we trust, all others must bring data