82,883
edits
Metric: Difference between revisions
no edit summary
Amwelladmin (talk | contribs) No edit summary |
Amwelladmin (talk | contribs) No edit summary |
||
| Line 12: | Line 12: | ||
Not a law of economics or sociology so much as a wry remark — professor Goodhart made it at a symposium in 1975 — that happens to pierce modern management orthodoxy through its heart. Thus it can both spur its own industry of academic work in sociology and [[systems theory]], and at the same time go ignored in the upper tiers of corporate management: | Not a law of economics or sociology so much as a wry remark — professor Goodhart made it at a symposium in 1975 — that happens to pierce modern management orthodoxy through its heart. Thus it can both spur its own industry of academic work in sociology and [[systems theory]], and at the same time go ignored in the upper tiers of corporate management: | ||
{{Quote|When a measure becomes a target, it ceases to be a good measure.}} | {{Quote|When a measure becomes a target, it ceases to be a good measure.}}People are smart and selfish. They will work any target you set to suit themselves. If you tax by number of windows, people will board up their windows.<ref>See James C. Scott’s epic {{Br|Seeing Like a State}}.</ref> | ||
===On dolphins, seagulls and opposable thumbs=== | ===On dolphins, seagulls and opposable thumbs=== | ||
| Line 27: | Line 27: | ||
====Regressive==== | ====Regressive==== | ||
Using a single metric as a proxy to measure “[[multivariate]]” phenomena that are driven by several factors. Here [[Simpson’s paradox]] is not your friend. Much | Using a single metric as a proxy to measure “[[multivariate]]” phenomena that are driven by several factors. Here [[Simpson’s paradox]] is not your friend. Much “social justice” — which we define as the wishful, if not wilful, tendency to boil complex socioeconomic phenomena down to simplistic moral propositions that even a dull fifth-former could understand, and only a dull fifth-former would fall for — stumbles into this trap. | ||
====Extremal==== | ====Extremal==== | ||
Many metrics are useful ''within a range'' corresponding to “normalcy”: call it “peacetime”, “normal operating conditions”, “[[business as usual]]” and similar platitudes — but break down, fail, or even reverse themselves in extremes or unusual cases beyond that range. Using [[normal distribution]]s of independent events to model non-dependent events with non-linear distributions — like, well, anything that involves human behaviour, such as a market — is especially fraught, because even where, 95% of the time, your metrics work fabulously, that 95% is exactly the range over which ''it doesn’t matter whether they work or not''. This is the time where things are operating as normal, behaving themselves, and ''not'' blowing up. | |||
The “use-case” for any metric in the first place, remember, is to warn about risk events. No-one needs a light on the dashboard saying “everything is fine”. A “heightened risk” metric that you can only rely on when there isn’t a heightened risk is a ''waste of trees''. | |||
====Causal==== | ====Causal==== | ||
Metrics fall for the old “[[correlation]] is not the same as [[causation]]” chestnut. In recent years, prompted we think by the [[difference engine]]’s emergence as the machine of choice for measuring things, we have given up on the idea of proving out causal chains. We are happy enough to rely on correlations. But correlations may be meaningful or spurious, and even where meaningful they give no idea which way the causal arrow flows. It may be true that people often buy ice-cream when they are wearing sunglasses, but handing out complimentary sunglasses will not improve ice-cream sales. | |||
====Adversarial==== | ====Adversarial==== | ||