Template:M intro technology metadata

The desktop

In 1973, Xerox’s Palo Alto Research Center released the “Alto”. This was the first personal computer equipped with a “graphical user interface” (GUI) — computing with pictures — instead of the traditional “character user interface”.

If potential users — bowler-hatted bureaucrats who didn’t use computers at all — were to be persuaded to give up their card catalogue systems, typing pools and reusable manila envelopes and instead stare at a screen all day, the system would need to look as familiar as possible.

To lessen the cognitive burden, Xerox came up with a visual metaphor. The Alto’s interface was modelled on a “desktop”. Instead of an impenetrable wall of green code and a flashing cursor, users were presented with a cartoonish depiction of a literal desktop and all its familiar iconography: documents, folders, a blotter, filing cabinets, in-trays, out-trays and even a dinky little waste-paper basket.

All designed to reassure the meatware — as fearful of incipient obsolescence then as it is now — that the change journey from the comfy old analogue world to the coming atomic age would not be so bad after all.

The spreadsheet

In 1979, Dan Bricklin and Bob Frankston created a new application for the Apple II computer. They called it “VisiCalc”. It was a grid of cells that you could input numbers and text into and then run calculations on by reference to cell coordinates. VisiCalc was the first spreadsheet program: a primitive ancestor to that beast we all now know and love as Microsoft Excel.

VisiCalc’s brilliant innovation was to separate the data you wanted to manipulate — the numbers and text in the cells — from the logical operations you wanted to manipulate them with — quasi-mathematical formulae — which referenced just the coordinates of the cells holding the data, not the data itself. You could therefore change the data without upsetting the calculation parameters. VisiCalc established a rudimentary form of programming language. A spreadsheet is a sort of programme. This may seem redolent of a smart contract, by the way. That is because it is. But let us not be distracted.

It might not have seemed much in 1979, but VisiCalc and its heirs would revolutionise business computing. While not nearly as intuitive as the Alto’s “desktop” — there was no graphic user interface or anything like that — VisiCalc was a much purer expression of what a personal computer could do. It promised even modest undertakings a powerful means of storing, augmenting, filtering, analysing and manipulating unprecedented amounts of information as structured data.

Good and bad metaphors

A spreadsheet is a much better way of thinking about how to organise digital information than a desktop because it is not constrained by physical space. Whereas the information on a traditional desktop is embedded in a physical “substrate” — usually paper — digital information has no such limitations. An empty spreadsheet stretches endlessly away in two directions:

Downwards: You can add items — “artefacts” — to your filing system without limit, unconstrained by the area of your desk or the volume of your filing cabinet. Each new artefact occupies a new row. There is an infinite number of rows.
Across: You can categorise each artefact however you like by creating new columns. There is no limit to the number of columns and no necessary hierarchy between them.

The desktop is designed to manage physical properties that digital information does not have. In printed information, paper is “form”, text is “substance”. A desktop must prioritise form because in a physical system substance cannot exist independently of form. Data must be printed on paper. Paper must be put somewhere. Unless you physically copy it, you can only put a piece of paper in one place at a time.

Furthermore, storing paper is expensive. Copying and transporting paper is expensive, slow and “lossy”. The embedded data degrades. Each copy increases storage costs at the Iron Mountain.

Whereas a desktop subfolder is necessarily a sub-division of the folder it sits in, spreadsheet columns have no hierarchy and need not bear any relation to each other, as long as they all relate to the original artefact.

But digital information has (almost) no form. It does not occupy physical space. It costs nothing to store. We can copy and move it costlessly, instantly, and with no loss of fidelity. At least when compared with physical information, digital information can be everywhere at once. We are not constrained by space or time when we store or move digital information. Yet to file it, we use a metaphor that assumes we are.

Division versus multiplication

In a “desktop” structure, subfolders are sub-divisions, each further level down more fine-grained and subordinate than the last, and less important relative to the formal hierarchy. A folder path is a rabbit hole. Hierarchy takes priority over the artefact. The hierarchy explains and contextualises everything. The more extensive the hierarchy, the less significant an individual artefact.

Whereas folders are divisive in nature, spreadsheet columns are multiplicative. All columns in a spreadsheet have equal standing — they are, well, pari passu — so they can be multiplied without limit: if an existing column, or an artful combination of them, doesn’t yield the information you need, you can always add more columns. In a spreadsheet, the artefact takes priority over the hierarchy. The hierarchy is incidental. Formal.

But naturally we like hierarchy. Hierarchies place objects in a permanent relationship to each other. This is comforting in a changing world. It feels tangible in a virtual realm that natively is not tangible. We naturally organise ourselves into hierarchies — for better or worse — so there is no surprise that we like our information served up with similar structure.

Flexibility is good for experts, virtuosi, and improvisers: it intimidates those who are not. Structure works better where we want dependability and reliability.

A front in the battle between substance and form

The desktop prioritises form.
A spreadsheet prioritises substance.

Here we find our old friend the struggle between form and substance: if we take it that, whatever your metaphor of choice, the “artefact” — the thing being filed — is the substance and the organising system it goes into is the form, we can see that the desktop and the spreadsheet have fundamentally opposed philosophies.

The desktop priorities form — the artefact is buried at the bottom of a rigid formal structure of folders and subfolders which, once created, cannot easily be altered. This is why it is so hard to find things you have misfiled. You cannot put anything into the database until you have fully specified its folder path.

By contrast, a spreadsheet prioritises substance. The artefact is the first thing to go in the database, before any formal structure is applied to it. It sits at the top of the structure. Only once it is in situ can you assign it any formal properties. The artefact therefore wears the properties we assign it lightly. Its position and identity does not change if we later alter, remove or augment the values we assign to it.

Metadata

Each desktop folder or spreadsheet column is metadata about its artefact — literally, “information about information”.

A folder structure generates a very narrow, wan sort of metadata in the form of folder names: they are limited to a finite number of text characters. It is so limited that it is hardly worth thinking of it as metadata at all. Indeed, the Windows operating system could, but doesn’t, treat folder names as metadata, which is mad.

A spreadsheet, by contrast, uses metadata powerfully, putting few limits on what form it can take: text, calculable numbers and dates, checkboxes, people, colours, flags, choices, lookups, comments, or calculations. It can be validated, managed, controlled, compulsory, optional, pre-populated or free-form. You can then filter, group, sort, chart, pivot and triangulate. Your imagination is the limit.

The more metadata you have, the more ways you can look at the data. Each separate value represents a new and distinct way of organising information. Even if the metadata is wrong, the inconsistencies between it and other correct fields on the same artefact allow us to triangulate and identify problematic data. We can, in this way, generate metadata about metadata. This is meta-metadata.

The desktop clings on

Yet even in our modern, hyper-networked, cloud-based work environment — even though we have had Microsoft Excel for nearly 40 years, the desktop metaphor hangs on.

We still call them “desktops”, though now for the prosaic reason that they generally are the only thing that sits on top of our desk. Look: the desktop was a nice, quaint idea. It got old geezers in green visors to sit down at keyboards — for that, the change managers of the world can be truly grateful — but the metaphor has well-outlived its purpose now.

It assumes each artefact to be filed has a unique physical location as if it were shelved in a physical library. Older readers may remember the Dewey decimal system, which numbered the entire corpus of non-fiction wisdom from zero to 1,000.

When information is digital and has no physical dimension this is an unnecessary constraint. Duplicating artefacts to suit multiple hierarchies creates basis risk. Which was the canonical version of the document? How can we be sure they are the same? What happens if one, but not the other, gets updated?

Where the document is a “living thing,” plotting its own miserable trajectory through the cosmos — say, a contract under negotiation, or a maintained legal template — then running multiple copies multiplies the job of maintaining all copies as the document changes, and that introduces the risk of human error. There may be miskeys. A document may be forgotten. Version control is a pain.

Also, a preferred hierarchy can change. Personnel, managers, business priorities, and circumstances change. They change the priorities of formal organisation. Changing your preferred hierarchy means completely re-engineering your folder structure.

Substrate neutrality

These are all problems of the physical realm; the spreadsheet metaphor shows us we need not be troubled by them in the digital realm. Here, the physical “substrate” — the hard copy — is irrelevant. What matters is the ASCII code embedded in it. In the digital realm, it has been abstracted and floats free of the substrate.

Across a diverse network of collaborators, the freedom to create multiple organising hierarchies on the fly, without upsetting other users, is immensely empowering.

Our reverence for the sacred substrate fell away, but not entirely. We still revere wet ink, for some reason counterparts clauses, and the dear old desktop.

For still, as we file, we cannot resist the siren call of folders. Folders in folders in folders in folders in folders.

Why do we persist with folders?

More than twenty years ago Tom Zingale taught a young JC a valuable lesson. Battling with some byzantine folder structure, and losing, JC cried out in anguish, and Tom said this:

JC: How on earth am I meant to organise all this?

Tom: With metadata.

JC: Er, with what?

Tom: Metadata. The answer to your question is metadata. Metadata, metadata, metadata. Whatever your question is, the answer is metadata.

JC: Well, my question is, “How do I use metadata to fix this filing problem?

Tom: Oh, right. Simple: SharePoint.

Wait: SharePoint?

About SharePoint

Now a lot of good people viscerally hate SharePoint. To be sure, Microsoft seems to have gone out of its way to foment this hatred. It seems to have conducted a programme, over 20 years, to make SharePoint as hard to love as it can.

But at the same time, Microsoft has rebuilt its entire Office 35 Suite around the SharePoint platform. It is, to be sure monumentally confusing, the Teams integration is baffling. The utterly dismal online versions of its Office suite drive people righteously up the wall.

But, still, a good part of the enmity for SharePoint arises from the users’ basic misunderstanding of SharePoint’s fundamental architecture.

SharePoint is the first, philosophically, digitally native operating system. It has abandoned the desktop metaphor. SharePoint uses the spreadsheet metaphor.

In SharePoint you organise by metadata, not by folders.

DO NOT USE FOLDERS IN SHAREPOINT. DO NOT COMPLAIN THAT SHAREPOINT SUCKS IF YOU USE FOLDERS. IF YOU USE FOLDERS IN SHAREPOINT, THAT MEANS YOU SUCK, NOT SHAREPOINT.

Folders are top-down. Metadata is bottom-up. Folders prefer form over substance. Metadata prefers substance over form.

SharePoint allows you to do exactly the same thing with a document library as Excel allows you to do with a spreadsheet.

So it is odd — isn’t it? We intuitively understand the power of metadata when we are presented with a spreadsheet. But the same power does not occur to us when we are presented with a file management system. The desktop metaphor is burned on our retina.

Even though it is, in essence, a supercharged online spreadsheet, SharePoint continues to be resented by almost everyone.