Template:M intro technology metadata

The desktop

In 1973, Xerox’s Palo Alto Research Center released the “Alto”. This was the first personal computer equipped with a “graphical user interface” (GUI) — computing with pictures — instead of the traditional “character user interface”.

If potential users — bowler-hatted bureaucrats who didn’t use computers at all — were to be persuaded to give up their card catalogue systems, typing pools and reusable manila envelopes and instead stare at a screen all day, the system would need to look as familiar as possible.

To lessen the cognitive burden, Xerox came up with a visual metaphor. The Alto’s interface was modelled on a “desktop”. Instead of an impenetrable wall of green code and a flashing cursor, users were presented with a cartoonish depiction of a literal desktop and all its familiar iconography: documents, folders, a blotter, filing cabinets, in-trays, out-trays and even a dinky little waste-paper basket.

All designed to reassure the meatware — as fearful of incipient obsolescence then as it is now — that the change journey from the comfy old analogue world to the coming atomic age would not be so bad after all.

The spreadsheet

In 1979, Dan Bricklin and Bob Frankston created a new application for the Apple II computer. They called it “VisiCalc”. It was a grid of cells that you could input numbers and text into and then run calculations on by reference to cell coordinates. VisiCalc was the first spreadsheet program: a primitive ancestor to that beast we all now know and love as Microsoft Excel.

VisiCalc’s brilliant innovation was to separate the data you wanted to manipulate — the numbers and text in the cells — from the logical operations you wanted to manipulate them with — quasi-mathematical formulae — which referenced just the coordinates of the cells holding the data, not the data itself. You could therefore change the data without upsetting the calculation parameters. VisiCalc established a rudimentary form of programming language. A spreadsheet is a sort of programme. This may seem redolent of a smart contract, by the way. That is because it is. But let us not be distracted.

It might not have seemed much in 1979, but VisiCalc and its heirs would revolutionise business computing. While not nearly as intuitive as the Alto’s “desktop” — there was no graphic user interface or anything like that — VisiCalc was a much purer expression of what a personal computer could do. It promised even modest undertakings a powerful means of storing, augmenting, filtering, analysing and manipulating unprecedented amounts of information as structured data.

Good and bad metaphors

A spreadsheet is a much better way of thinking about how to organise digital information than a desktop because it is not constrained by physical space. Whereas the information on a traditional desktop is embedded in a physical “substrate” — usually paper — digital information has no such limitations. An empty spreadsheet stretches endlessly away in two directions:

Downwards: You can add items — “artefacts” — to your filing system without limit, unconstrained by the area of your desk or the volume of your filing cabinet. Each new artefact occupies a new row. There is an infinite number of rows.
Across: You can categorise each artefact however you like by creating new columns. There is no limit to the number of columns and no necessary hierarchy between them.

The desktop is designed to manage physical properties that digital information does not have. In printed information, paper is “form”, text is “substance”. A desktop must prioritise form because in a physical system substance cannot exist independently of form. Data must be printed on paper. Paper must be put somewhere. Unless you physically copy it, you can only put a piece of paper in one place at a time. Older readers may remember the Dewey decimal system, which numbered the entire corpus of non-fiction wisdom from zero to 1,000. This addressed an exclusively physical problem. We don’t need the Dewey system any more. We don’t need the desktop, either.

Furthermore, storing paper is expensive. Copying and transporting paper is expensive, slow and “lossy”. The embedded data degrades. Each copy increases storage costs at the Iron Mountain.

Whereas a desktop subfolder is necessarily a sub-division of the folder it sits in, spreadsheet columns have no hierarchy and need not bear any relation to each other, as long as they all relate to the original artefact.

But digital information has (almost) no form. It does not occupy physical space. It costs nothing to store. We can copy and move it costlessly, instantly, and with no loss of fidelity. At least when compared with physical information, digital information can be everywhere at once. We are not constrained by space or time when we store or move digital information. Yet to file it, we use a metaphor that assumes we are.

Division versus multiplication

In a “desktop” structure, subfolders are sub-divisions, each further level down more fine-grained and subordinate than the last, and less important relative to the formal hierarchy. A folder path is a rabbit hole. Hierarchy takes priority over the artefact. The hierarchy explains and contextualises everything. The more extensive the hierarchy, the less significant an individual artefact.

Whereas folders are divisive in nature, spreadsheet columns are multiplicative. All columns in a spreadsheet have equal standing — they are, well, pari passu — so they can be multiplied without limit: if an existing column, or an artful combination of them, doesn’t yield the information you need, you can always add more columns. In a spreadsheet, the artefact takes priority over the hierarchy. The hierarchy is incidental. Formal.

But naturally we like hierarchy. Hierarchies place objects in a permanent relationship to each other. This is comforting in a changing world. It feels tangible in a virtual realm that natively is not tangible. We naturally organise ourselves into hierarchies — for better or worse — so there is no surprise that we like our information served up with similar structure.

Flexibility is good for experts, virtuosi, and improvisers: it intimidates those who are not. Structure works better where we want dependability and reliability.

A front in the battle between substance and form

The desktop prioritises form.
A spreadsheet prioritises substance.

Here we find our old friend the struggle between form and substance: if we take it that, whatever your metaphor of choice, the “artefact” — the thing being filed — is the substance and the organising system it goes into is the form, we can see that the desktop and the spreadsheet have fundamentally opposed philosophies.

The desktop priorities form — the artefact is buried at the bottom of a rigid formal structure of folders and subfolders which, once created, cannot easily be altered. This is why it is so hard to find things you have misfiled. You cannot put anything into the database until you have fully specified its folder path.

By contrast, a spreadsheet prioritises substance. The artefact is the first thing to go in the database, before any formal structure is applied to it. It sits at the top of the structure. Only once it is in situ can you assign it any formal properties. The artefact therefore wears the properties we assign it lightly. Its position and identity does not change if we later alter, remove or augment the values we assign to it.

Metadata

Each desktop folder or spreadsheet column is metadata about its artefact — literally, “information about information”. A folder structure generates a limited, anaemic sort of metadata in the shape of folder 260-character labels: so limited that the Windows operating system does not treat folder names as metadata at all. A spreadsheet, by contrast, imposes few limits on what form metadata can take: text, calculable numbers and dates, checkboxes, people, colours, flags, choices, lookups, comments, or calculations. It can be validated, managed, controlled, compulsory, optional, pre-populated or free-form. You can filter, group, sort, chart, or pivot it.

The more metadata you have, the more ways you can look at your data. The more ways you can access your data. Each separate value represents a new and distinct way of organising information. Even if your metadata is wrong, you can triangulate the inconsistencies to identify problematic data. You can, in this way, generate metadata about metadata. This is meta-metadata.

The desktop clings on

Yet even in our modern, hyper-networked, cloud-based work environment — even though we have had Microsoft Excel for nearly 40 years, the desktop metaphor hangs on. We still call them “desktops”, for the prosaic reason that they are the only thing still allowed on the desktop in our clear-desk, humans-as-fungible-cogs-in-the-machine modern office environment. (Is it any wonder firms are struggling to get staff back to the office, by the way?)

The desktop was a nice, quaint idea. It got old geezers in green visors to sit down at keyboards. For that, the change managers of the world can be grateful, but the metaphor has long since outstayed its welcome now. Enough already of the dinky desktop.

When information is digital and has no physical dimension it is an unnecessary constraint. Duplicating artefacts to suit multiple hierarchies creates basis risk. Which was the canonical version of the document? How can we be sure they are the same? What happens if one, but not the other, gets updated?

Where the document is a “living thing,” plotting its own miserable trajectory through the cosmos — say, a contract under negotiation, or a maintained legal template — then running multiple copies multiplies the job of maintaining all copies as the document changes, and that introduces the risk of human error. There may be miskeys. A document may be forgotten. Version control is a pain.

Also, a preferred hierarchy can change. Personnel, managers, business priorities, and circumstances change. They change the priorities of formal organisation. Changing your preferred hierarchy means completely re-engineering your folder structure.

Substrate neutrality

These are all problems of the physical realm; the spreadsheet metaphor shows us we need not be troubled by them in the digital realm. Here, the physical “substrate” — the hard copy — is irrelevant. What matters is the ASCII code embedded in it. In the digital realm, it has been abstracted and floats free of the substrate.

Across a diverse network of collaborators, the freedom to create multiple organising hierarchies on the fly, without upsetting other users, is immensely empowering.

Our reverence for the sacred substrate fell away, but not entirely. We still revere wet ink, for some reason counterparts clauses, and the dear old desktop.

For still, as we file, we cannot resist the siren call of folders. Folders in folders in folders in folders in folders.

Why do we persist with folders?

More than twenty years ago Mr T. Zingale taught the young JC a valuable lesson. Battling with some byzantine folder structure, and losing, JC cried out in anguish:

JC: How on earth am I meant to organise all this?

Mr. T: With metadata.

JC: Er, with what?

Mr. T: Metadata. The answer to your question is metadata. Metadata, metadata, metadata. Whatever your question is, the answer is metadata.

JC: Well, my question is, “How do I use metadata to fix this filing problem?

Mr. T: Oh, right. Simple: SharePoint.

Wait: did somebody say, SharePoint?

About SharePoint

I know I’m unlovable
You don’t have to tell me.
Message received loud and clear
[...]
Loud and clear
If I seem a little strange well
That’s because I am.

—The Smiths, Unlovable

Now a lot of good people viscerally hate SharePoint. To be sure, Microsoft has apparently gone out of its way to foment this emotion, waging a sustained campaign, over 20 years, to render it as unlovable as can be.

But still, SharePoint was the first — and remains the only — philosophically digital operating system. It abandoned the dinky desktop metaphor and embraced the spreadsheet approach. Think of it as an online version of Excel that can also hold documents in a spreadsheet format. You can create as many new metadata columns about each document, just like you can in Excel.

And Microsoft must realise this: it has rebuilt its entire productivity suite around SharePoint. To be sure, the Office 365 platform — or whatever they call it now — is monumentally confusing: the Teams integration is baffling. The interaction between OneDrive, Teams, File Explorer, SharePoint and Outlook seems wilfully designed to create as much confusion, duplication and data loss as possible. Where once there was a single place for documents, communications and schedules, now there are three or four of each. Did you get that file I sent? I chatted it to you in Teams. Or, hang on, did I email it? Or Share a link from OneDrive?

Disaster.

The unbearable tolerance of spreadsheets

And this is the downside of the spreadsheet metaphor. A spreadsheet is a blank page: it leaves hierarchy and database design up to whoever wants to have a go at it. SharePoint leaves the discipline of getting the database design right, and then enforcing it, up to the user.

But this pluralism, this multiplicity, this total ambivalence to how users want to work, this tolerance of all the different possible ways of working, however dopey, this total disarming agnosticism — that will even let users embrace the desktop metaphor if that’s what they really want — is a recipe for organisational chaos.

Because, as we should know by now, most users will embrace the desktop metaphor if you give them a chance. Because they don’t know any better. Why would they? Database design optimisation is its own branch of computer engineering.

Most users do not have a certificate in database design and management. Nor do they want one. They have not thought much about database optimisation.

Rather, when they entered the workforce, or first clapped eyes on a computer, they saw a desktop and they liked it. They understood it and got very used to it.

The metaphor has proven resilient. It will remain hard to dislodge.

So it is odd — isn’t it? We intuitively understand the power of metadata when we are presented with a spreadsheet. But the same power does not occur to us when we are presented with a file management system. The desktop metaphor is burned on our retina.

Even though it is, in essence, a supercharged online spreadsheet, SharePoint continues to be resented by almost everyone.

This is no more than Microsoft deserves for is terrible implementation, but at least give it a try. Cross the Rubicon. Redpill yourself. Leave the desktop and go truly digital.