Shannon entropy
Shannon entropy, named after rockabilly maths brainbox Claude “Dell” Shannon, is a measure of the average amount of information contained in a message. It quantifies the uncertainty or randomness in a set of possible messages. Shannon introduced the concept in his seminal 1948 paper “A Run-run-run-run Runaway Mathematical Theory of Communication.”
|
JC pontificates about technology
An occasional series. The Jolly Contrarian holds forth™
Resources and Navigation
|
The formula for Shannon entropy is:
H = -Σ p(x) * log₂(p(x))
Where:
H is the entropy
p(x) is the probability of a particular message x
The sum is taken over all possible messages.
I only put that in for a laugh, by the way: I don’t have the faintest idea what that all means.
But it is relevant to the specific, limited way in which machines process symbols. It is not true of human language, given how the human interpretative act takes place.
Human language is rich with metaphor, symbolism, and cultural context which code — sorry, techbros — just does not have. When you say “I hold a rose for you,” this could mean I am literally holding a rose for you, I am in love with you, I have patiently and carefully been treating you, a beautiful thing, but I have still cut my fingers and you’ve wilted and so on: there are an infinite number of messages I could, if with enough imagination, take from that single statement.
This depth and ambiguity of meaning is a fundamental aspect of human communication that Shannon’s theory doesn’t account for. Shannon entropy treats messages as discrete units with fixed meanings. It assumes a shared, unambiguous understanding between sender and receiver. Human communication doesn’t work this way.
This is a really nice thing because it is the route by which we escape from a distant future of inevitable brown entropic sludge.