Big data

The JC’s amateur guide to systems theory™ Go Premium Index: Click ᐅ to expand: Systems theory The Devil’s Advocate Comments? Questions? Suggestions? Requests? Insults? We’d love to 📧 hear from you. Sign up for our newsletter.

As at the time of its analysis, all data is from the past.

—Roger Martin

Turkey: “I have transformed myself into a data-driven business. All my data — and I’ve got reams of the stuff — tells me that every morning I shall be fed at 9 am on the dot. Aha! Here comes the farmer, right on time! I wonder if I’ll get a special treat because it is Christmas!”

Charlotte (spinning web): Ummm

Beware an over-commitment to data analytics:

It is premium mediocre

Firstly it expresses a preference for the aggregate over the specific, and the average over the outlier, the individual, the unique or extraordinary. It is to prefer the mediocre, for its weight of numbers, over the isolated vision of a genius or the depravity of the ugliest man.

As surely as the ugliest man killed God, so did data kill the superman. The will to power is defeated by the million-strong dull blades of the will to entropy. It is the will to premium mediocre.

It is historical

All data is from the past, as Roger Martin has it. That means all data is skewed in time

It is bad

Not only (per our careful argument at signal-to-noise ratio) is the overall quantity of data we have skewed in time (all from the past, none from the future), place (only what we’ve been looking at, none of what we haven’t) and practically nil in quantity, the quality of data in our tawdry collection is poor. And not just in its profusion of cat videos and hot takes on Twitter, either. For, as an evolutionary record, it contains all the errors and the one successful trial; all the abandoned drafts, all the false starts, all the typos, split infinitives, tendentious arguments, feeble caveats and needless avoidances of doubt. The data we have, that is, even on our own rationalised terms, mainly noise. Bad noise.

Noise

There are two kinds of noise.

There is good noise: random, valid information that is just not the information you are looking for, but is broadcast on the same frequency (background radiation, cosmic chatter, cocktail party hubbub — hundreds of other perfectly meaningful exchanges; just not the one you are interested in) and there is bad noise: errors, mistakes, hot takes on Twitter and so on.

And we self-select for bad noise, eschewing good noise as being worthless: scientific journals do not publish accounts of research programmes that turn out badly. Little evidence remains in the fossil record of the failed experiments, unsuccessful iterations and occasions where, under laboratory conditions, the data did not do what the experimenter hoped they would, for no-one publicises non-results. Even for the good experiments, the published research that it makes it past peer review is filtered and cleansed of all those false starts and misconceptions that led to the Eureka moment. Mean time, the hot takes on Twitter, conspiracy theories and cat videos continue to mushroom.

It is illiberal

Second, in its reductionism, in its funnelling of a dispersed population into an essential homogeneity, it speaks to the underlying belief in a grand unifying theory of everything: a transcendent truth. This, in the JC’s view, is a profoundly illiberal idea: to be unable to accommodate pluralism is to deny of pluralism.

It is noisy

Thirdly, to embrace all the data you can find is to degrade the signal-to-noise ratio. Even if you buy into the incoherent reductionist idea that the “signal” is some kind of transcendent truth, by industrialising your data, you risk burying it and if you don’t — if like we pluralists you see any signal as not just a suitable narrative for your present purposes, the more data you gather, the more possible narratives — conflicting narratives; incommensurable narratives — you will have. Now this is, for a pluralist, is a good thing: every narrative is a tool in your workshop, the more you have the better you are equipped to deal with the unknown unknowns our complex world will surely throw at us — but that tends not to be what big data disciples are after.

It is not a universal affirmative

Even if, from pure data, you could establish the causal relationship between data you have observed and an event that drives it (it is axiomatic that you can’t, by the way: you can only derive a correlation, and we know how spurious those can be) you still can’t conclude that the cause propelling the general is the same one that compelled any particular.

Averages are crappy things to aspire to, or configure your business to, for a number of reasons.

Because the machinations of statistics can, in certain contexts, inflame the passions of the righteous, the JC has devised the parable of the squirrels to tease this out.

See also

• Signal-to-noise ratio
• In God we trust, all others must bring data