Desktops, metadata and filing: Difference between revisions

From The Jolly Contrarian
Jump to navigation Jump to search
No edit summary
No edit summary
Line 15: Line 15:


====Why it’s is a better metaphor====
====Why it’s is a better metaphor====
{{drop|A|spreadsheet is}} a ''much'' better way of thinking about how to organise digital information than a desktop because digital information has no physical dimension. It is not constrained by the physical substrate — usually paper — that analogue information is embedded in, and desktops and filing cabinets were designed around.  
{{drop|A|spreadsheet is}} a ''much'' better way of thinking about how to organise digital information than a desktop because digital information has no physical dimension. It is not constrained by a physical “[[substrate]]” — usually paper — that analogue information is embedded in.


In printed information, the paper is the form, the text in it is the substance. An analogue filing system is obliged to prioritise form over substance.
The desktop was designed around that physical problem: how to manage bits of paper. In printed information, paper is “form”, text is “substance”. A desktop is obliged to prioritise form over substance because the substance ''does not exist independently of the form''.


Analogue information must ''put'' somewhere and, unless you copy the physical form, you can only put it in one place. Copying and then transporting physical information is expensive, slow and “lossy”. Each time we copy, we lose fidelity and increase our storage costs. All physical information is eventually destined for the [[Iron Mountain]].
Paper must be ''put'' somewhere. Unless you physically copy it, it can only be in one place.  


Digital information has no form. It does not occupy physical space. It costs nothing to store.<ref>Okay: ''almost'' nothing.</ref> We are not constrained by three-dimensional space when we store or move it. We move it by copying it, and it moves instantly, and with no loss of fidelity. At least when compared with physical information, digital information can be everywhere at once. 
Copying and transporting paper is expensive, slow and “lossy”. Each copy loses fidelity and increases storage costs.<ref>All physical information is eventually destined for the [[Iron Mountain]].</ref>


The spreadsheet metaphor is not constrained by the physical limitations of a desktop.
Digital information has (almost) no form.<ref>Okay: ''almost'' no form. Compared with physical information. In this section take the word “almost” as read. </ref> It does not occupy physical space. It costs nothing to store. We can copy and move it costlessly, instantly, and with no loss of fidelity. At least when compared with physical information, digital information can be everywhere at once.  We are not constrained by space or time when we store or move digital information. Yet to file it, we use a metaphor that assumes we are.


Being a ''conceptually'' infinite number of rows and columns, a spreadsheet extends in two infinite directions: {{L1}}'''Downwards''': You can ''add'' items to your filing system without limit, unconstrained by the area of your desk or the volume of your filing cabinet, where each item occupies one of an infinite number of rows. <li>'''Across''': You can ''categorise'' each item in an unlimited number of ways by creating a new ''column''. There is no relative hierarchy between columns. They need not even bear any relation to each other as long as they relate to the original item. Whereas a subfolder is necessarily a sub-division of the folder it sits in, this is not true of a new column. </ol>
A spreadsheet is not constrained by the physical limitations of a desktop: being a ''conceptually'' infinite number of rows and columns, a spreadsheet extends in two infinite directions: {{L1}}'''Downwards''': You can ''add'' items to your filing system without limit, unconstrained by the area of your desk or the volume of your filing cabinet, where each item occupies one of an infinite number of rows. <li>'''Across''': You can ''categorise'' each item in an unlimited number of ways by creating a new ''column''. There is no relative hierarchy between columns. They need not even bear any relation to each other as long as they relate to the original item. Whereas a subfolder is necessarily a sub-division of the folder it sits in, this is not true of a new column. </ol>
====Division versus multiplication====
====Division versus multiplication====
{{drop|I|n a “desktop”}} structure, subfolders are sub-''divisions'' of a folder, each further level down the folder structure becoming more fine-grained and subordinate, and less important relative to the formal hierarchy.  
{{drop|I|n a “desktop”}} structure, subfolders are sub-''divisions'', each further level down more fine-grained and subordinate than the last, and less important relative to the formal hierarchy.  We prioritise the ''hierarchy over the item''. The hierarchy explains and contextualises everything.


All columns in a spreadsheet have equal standing — they are, well, ''[[pari passu]]'' — and their combination has a ''multiplicative'' effect: if an existing column, or an artful combination of columns, doesn’t yield the information you need, you can always add more columns.  
All columns in a spreadsheet have equal standing — they are, well, ''[[pari passu]]'' — and their combination has a ''multiplicative'' effect: if an existing column, or an artful combination of columns, doesn’t yield the information you need, you can always add more columns.  In a spreadsheet, we prioritise the ''item over the hierarchy''. The hierarchy is incidental.
====A front in the battle between substance and form====
====A front in the battle between substance and form====
{{quote|
{{quote|
Line 42: Line 42:
{{drop|E|ach folder or}} column is its own item of [[Metadata|''metadata'']] ''—'' literally, “information ''about'' information” — about the item being filed.<ref>Grammar pedants’ corner: Even though “data” is plural, “metadata” is generally treated as a singular mass noun. Please direct your letters to the Royal Statistical Society — not because it is their fault: rather, they might keep metadata about this sort of thing.</ref>  
{{drop|E|ach folder or}} column is its own item of [[Metadata|''metadata'']] ''—'' literally, “information ''about'' information” — about the item being filed.<ref>Grammar pedants’ corner: Even though “data” is plural, “metadata” is generally treated as a singular mass noun. Please direct your letters to the Royal Statistical Society — not because it is their fault: rather, they might keep metadata about this sort of thing.</ref>  


A desktop folder structure can use a very limited range metadata: a limited number of characters of text. So limited that it is hardly worth thinking of it as metadata at all.<ref>Indeed, the Windows operating system doesn’t treat folder names as metadata at all, which is mad.</ref>
A hierarchical folder structure uses a very limited range of metadata: just characters of text, and a limited number at that: so limited that it is hardly worth thinking of it as metadata at all.<ref>Indeed, the Windows operating system doesn’t treat folder names as metadata at all, which is mad.</ref>


In a spreadsheet, by contrast, metadata can take the form of text, calculable numbers and dates, checkboxes, people,<ref>As in, a lookup to an object in a people directory, and not just a text name.</ref> colours, flags, choices, lookups, comments, or calculations. It can be validated, managed, controlled, compulsory, optional, pre-populated or free-form.
A spreadsheet, by contrast, puts no limits on how we use metadata. It can take the form of text, calculable numbers and dates, checkboxes, people,<ref>As in, a lookup to an object in a people directory, and not just a text name.</ref> colours, flags, choices, lookups, comments, or calculations. It can be validated, managed, controlled, compulsory, optional, pre-populated or free-form.
Each extra piece of metadata enriches the existing data in the row without detracting from it. Indeed, even if the metadata is wrong, the inconsistencies between the fields allow a user to triangulate and identify likely wrong — or problematic — material.


Metadata is, in this way, “non-destructive”. It only augments. Each metadata field creates its own way of ordering information. Each is its own hierarchy. If metadata is non hierarchical, it can be much more specific — you can date-stamp to the second, where you might have had a folder for a month.
Each extra piece of metadata enriches the item without detracting from it. Even if the metadata is wrong, the inconsistencies between it and other correct fields allow us to triangulate and identify problematic data.
 
Metadata is, in this way, “non-destructive”. It only augments. Each metadata field creates a new way of ordering information. If not being used to force a hierarchy, it can be much more detailed — you can date-stamp your item to the microsecond, where you might have had a folder for a month.


Suddenly, you can organise the same information in multiple different ways, simultaneously, without upsetting anyone else’s existing categorisation.
Suddenly, you can organise the same information in multiple different ways, simultaneously, without upsetting anyone else’s existing categorisation.
Line 57: Line 58:
This is ''meta''[[metadata]].
This is ''meta''[[metadata]].


The spreadsheet approach to file management is, thus “multi-hierarchical” and non-destructive. What about unused metadata? Ignore it. Unused hierarchies are almost costless. And you just never know —
The “spreadsheet” approach to file management is hierarchy agnostic (or “multi-hierarchical”, if you like things to have hierarchies). What about unused metadata? Ignore it. Unused hierarchies are almost costless. And you just never know —


==== The desktop clings on ====
==== The desktop clings on ====
Yet on our modern, hyper-networked, cloud-based work environment the desktop [[metaphor]] hangs on. We still call them “desktops”, though now for the prosaic reason that they generally are the only thing that sits on top of our desk. The desktop was a nice, quaint idea, and it got old men in green visors to sit down at a keyboard, and for that the ranks of middle management can be truly grateful, but it has well-outlived its purpose now.
{{drop|Y|et even in}} our modern, hyper-networked, cloud-based work environment — even though we have had Microsoft Excel for nearly 40 years, the desktop [[metaphor]] hangs on.  
 
Because ''physical'' information that sits on a real desktop, and ''digital'' information that sits on a computer are very different [[Ontology|ontological]] propositions.
 
The desktop metaphor asks us to put our files in folders, as we would do on a real desk. If a folder gets too big, we create subfolders. And, just as with a real desk, once we have put a file in one folder, we can’t very well put it anywhere ''else''. Just as with a real filing cabinet, if we misfile our subfolder, we might never find it again.
 
In the real world of physical information, that does no more than reflect grim corporeal reality: a thing can only be in one place at one time so that’s that. If the boss wants to file by customer, and you want to file by industry, then tough.
 
Physical filing systems reflect this: there is a unique physical location for any single document. So do physical filing ''methodologies'': older readers may remember the [[Dewey decimal system]], which numbered the entire corpus of non-fiction wisdom from zero to 1,000.<ref>My favourite was [[001.9]].</ref>
 
If the same document does need to be categorised in different ways — say the legal department needs to file by customer and the credit department by industry, this could only be achieved by ''duplicating'' the document and holding one version in each location. Legal would have a filing system, and credit would have another.


Plainly, this is an imperfect state of affairs. It created a [[Basis|''basis'']] ''risk''. Which was the canonical version of the document? What happened if one of them, but not the other, was updated? Where it is a “living thing” plotting its own miserable trajectory through the cosmos — a contract under negotiation, or a periodically updated legal template, for example — then ''duplicating it'' is a ''bummer''. It duplicates the manual task of updating all copies of the document as it changes, and that introduces the opportunity for human error. There may be miskeys. A document may be forgotten. Version control is a pain.
We still call them “desktops”, though now for the prosaic reason that they generally are the only thing that sits on top of our desk. The desktop was a nice, quaint idea, and it got old men in green visors to sit down at a keyboard, and for that the ranks of middle management can be truly grateful, but it has well-outlived its purpose now.


In the physical realm, duplication was slow, imperfect and expensive and so, limited. At the time this seemed to be a drawback; with hindsight, it appears a valuable discipline.
The desktop assumes there is a unique physical location for any document as if it were in a physical library. Older readers may remember the [[Dewey decimal system]], which numbered the entire corpus of non-fiction wisdom from zero to 1,000.<ref>My favourite was [[001.9]].</ref>


Also a preferred hierarchy can ''change'', as personnel, managers, business priorities, or circumstances change. Changing your preferred hierarchy means ''completely re-engineering your folder structure''. It would be lovely to not have to do that every time there is a corporate reorganisation.
Plainly, this is an imperfect state of affairs. It creates a [[Basis|''basis'']] ''risk''. Which was the canonical version of the document, if there are duplicates? How can we be sure they are the same? What happens if one of them, but not the other, gets updated? Where the document is a “living thing” plotting its own miserable trajectory through the cosmos — a contract under negotiation, or a periodically updated legal template, for example — then ''duplicating it'' duplicates the manual task of updating all copies of the document as it changes, and that introduces the opportunity for human error. There may be miskeys. A document may be forgotten. Version control is a pain.


So: a folder structure assumes a ''single'' hierarchy and multiple copies of each item.
Also a preferred hierarchy can ''change'', as personnel, managers, business priorities, or circumstances change. Changing your preferred hierarchy means ''completely re-engineering your folder structure''.


==== Substrate neutrality ====
==== Substrate neutrality ====
These are all problems of the physical realm; the spreadsheet metaphor shows us we need not be so troubled in a digital realm. In the digital world, the ''physical'' “[[substrate]]” of a document — the paper it is made out of — is, to all intents and purposes, ''irrelevant''. What matters is the ASCII code embedded in that document. In a digital world it has been abstracted from the substrate and floats free. Within a [[diverse]] network of collaborators, this is immensely empowering.
{{drop|T|hese are all}} problems of the physical realm; the spreadsheet metaphor shows us we need not be so troubled in the digital realm. In the digital world, the ''physical'' “[[substrate]]” is ''irrelevant''. What matters is the ASCII code embedded in that document. In the digital realm, it has been abstracted from the substrate and floats free. Within a [[diverse]] network of collaborators, this is immensely empowering.


It did not take people long to realise that ''email was amazing''.
It did not take people long to realise that ''email was amazing''.

Revision as of 14:22, 28 September 2024

JC pontificates about technology
An occasional series.
What you see is what you get, yesterday
The Xerox PARC Alto “desktop”
VisiCalc on the Apple II
Index: Click to expand:
Tell me more
Sign up for our newsletter — or just get in touch: for ½ a weekly 🍺 you get to consult JC. Ask about it here.

A bad metaphor: the desktop

In 1973, Xerox’s Palo Alto Research Center released the “Alto” personal computer. This was the first machine to boast a graphical user interface (GUI) instead of the traditional character user interface.[1]

To lessen the cognitive burden on users — at the time, bowler-hatted civil servants, sleeve-gartered clerks and others whose mental framework comprised girls in a typing pool and boys running memoranda between office in-trays in reusable manila envelopes, and whose idea of “information technology” was a pneumatic tube system that launched invoices around the clanking pipes of the organisation like mortar bombs — Xerox PARC created the metaphor of the “desktop”.

If they were going to be asked to stare instead at a computer screen then best make it as familiar as possible. Thus, the desktop: not an impenetrable wall of green code and a flashing cursor, but a cartoonish depiction of a literal desktop, with its comforting iconography of manila folders, a blotter, filing cabinets, in-trays, out-trays and even a dinky little waste-paper basket.

All very familiar; all just so: perhaps the transition to the atomic age would not be so bad after all.

A better metaphor: the spreadsheet

In 1979, Dan Bricklin and Bob Frankston created a new application for the Apple II computer. They called it “VisiCalc”. It was the first spreadsheet program. VisiCalc was, of course, the ancient ancestor of that beast we all now know and love as Microsoft Excel.

It might not have seemed much in 1979, but it would revolutionise business computing. While not nearly as intuitive as the “desktop” — there was no graphic user interface or anything like that — VisiCalc was a much purer expression of what a personal computer could do. It promised even modest undertakings a powerful means of storing, augmenting, filtering, analysing and manipulating unprecedented amounts of information as structured data.

Why it’s is a better metaphor

Aspreadsheet is a much better way of thinking about how to organise digital information than a desktop because digital information has no physical dimension. It is not constrained by a physical “substrate” — usually paper — that analogue information is embedded in.

The desktop was designed around that physical problem: how to manage bits of paper. In printed information, paper is “form”, text is “substance”. A desktop is obliged to prioritise form over substance because the substance does not exist independently of the form.

Paper must be put somewhere. Unless you physically copy it, it can only be in one place.

Copying and transporting paper is expensive, slow and “lossy”. Each copy loses fidelity and increases storage costs.[2]

Digital information has (almost) no form.[3] It does not occupy physical space. It costs nothing to store. We can copy and move it costlessly, instantly, and with no loss of fidelity. At least when compared with physical information, digital information can be everywhere at once. We are not constrained by space or time when we store or move digital information. Yet to file it, we use a metaphor that assumes we are.

A spreadsheet is not constrained by the physical limitations of a desktop: being a conceptually infinite number of rows and columns, a spreadsheet extends in two infinite directions:

  1. Downwards: You can add items to your filing system without limit, unconstrained by the area of your desk or the volume of your filing cabinet, where each item occupies one of an infinite number of rows.
  2. Across: You can categorise each item in an unlimited number of ways by creating a new column. There is no relative hierarchy between columns. They need not even bear any relation to each other as long as they relate to the original item. Whereas a subfolder is necessarily a sub-division of the folder it sits in, this is not true of a new column.

Division versus multiplication

In a “desktop” structure, subfolders are sub-divisions, each further level down more fine-grained and subordinate than the last, and less important relative to the formal hierarchy. We prioritise the hierarchy over the item. The hierarchy explains and contextualises everything.

All columns in a spreadsheet have equal standing — they are, well, pari passu — and their combination has a multiplicative effect: if an existing column, or an artful combination of columns, doesn’t yield the information you need, you can always add more columns. In a spreadsheet, we prioritise the item over the hierarchy. The hierarchy is incidental.

A front in the battle between substance and form

The desktop prioritises form.
A spreadsheet prioritises substance.

The last thing to notice is our old friend the struggle between form and substance: if we take it that, whatever your metaphor of choice, the “item” — the thing being filed — is the substance and the organising system it goes into is the form, we can see that the desktop and the spreadsheet have fundamentally opposed philosophies.

The desktop priorities form over substance — the “item” is buried at the bottom of a rigid formal structure of folders and subfolders which cannot easily be altered. This is why it is so hard to find things you have misfiled. You cannot put anything into the database until you have fully specified the folder path that comprises its formal structure.

By contrast, a spreadsheet prioritises substance over form. The “item” is the first thing to go in the database, naked of any formal structure. It therefore sits at the top of the structure. Only once it is in situ can you assign it any formal metadata properties. The item wears its metadata lightly, and is not affected if the metadata is later altered, removed or augmented.

Metadata

Each folder or column is its own item of metadata literally, “information about information” — about the item being filed.[4]

A hierarchical folder structure uses a very limited range of metadata: just characters of text, and a limited number at that: so limited that it is hardly worth thinking of it as metadata at all.[5]

A spreadsheet, by contrast, puts no limits on how we use metadata. It can take the form of text, calculable numbers and dates, checkboxes, people,[6] colours, flags, choices, lookups, comments, or calculations. It can be validated, managed, controlled, compulsory, optional, pre-populated or free-form.

Each extra piece of metadata enriches the item without detracting from it. Even if the metadata is wrong, the inconsistencies between it and other correct fields allow us to triangulate and identify problematic data.

Metadata is, in this way, “non-destructive”. It only augments. Each metadata field creates a new way of ordering information. If not being used to force a hierarchy, it can be much more detailed — you can date-stamp your item to the microsecond, where you might have had a folder for a month.

Suddenly, you can organise the same information in multiple different ways, simultaneously, without upsetting anyone else’s existing categorisation.

You can then filter and group your items by one or more columns. You can sort, chart, pivot and triangulate. The more metadata you have, the more ways you can look at the data.

You can even sort your data using data about how much metadata there is.

This is metametadata.

The “spreadsheet” approach to file management is hierarchy agnostic (or “multi-hierarchical”, if you like things to have hierarchies). What about unused metadata? Ignore it. Unused hierarchies are almost costless. And you just never know —

The desktop clings on

Yet even in our modern, hyper-networked, cloud-based work environment — even though we have had Microsoft Excel for nearly 40 years, the desktop metaphor hangs on.

We still call them “desktops”, though now for the prosaic reason that they generally are the only thing that sits on top of our desk. The desktop was a nice, quaint idea, and it got old men in green visors to sit down at a keyboard, and for that the ranks of middle management can be truly grateful, but it has well-outlived its purpose now.

The desktop assumes there is a unique physical location for any document as if it were in a physical library. Older readers may remember the Dewey decimal system, which numbered the entire corpus of non-fiction wisdom from zero to 1,000.[7]

Plainly, this is an imperfect state of affairs. It creates a basis risk. Which was the canonical version of the document, if there are duplicates? How can we be sure they are the same? What happens if one of them, but not the other, gets updated? Where the document is a “living thing” plotting its own miserable trajectory through the cosmos — a contract under negotiation, or a periodically updated legal template, for example — then duplicating it duplicates the manual task of updating all copies of the document as it changes, and that introduces the opportunity for human error. There may be miskeys. A document may be forgotten. Version control is a pain.

Also a preferred hierarchy can change, as personnel, managers, business priorities, or circumstances change. Changing your preferred hierarchy means completely re-engineering your folder structure.

Substrate neutrality

These are all problems of the physical realm; the spreadsheet metaphor shows us we need not be so troubled in the digital realm. In the digital world, the physicalsubstrate” is irrelevant. What matters is the ASCII code embedded in that document. In the digital realm, it has been abstracted from the substrate and floats free. Within a diverse network of collaborators, this is immensely empowering.

It did not take people long to realise that email was amazing.

From, more or less, a standing start in about 1993 — by lucky coincidence the year JC entered the workforce — the corporate world fell head over heels in love with electronic communication. Whatever reverence it had for the sacred substrate fell quickly away.[8] The expression, “this document is not worth the paper it is written on” has lost its meaning because the paper it is written no longer has much value at all.

Now we recognise the digital content embedded in the substrate is the valuable bit; the paper bit is just annoying. It is an inconvenient reminder of our erstwhile physical analogue reality. The better metaphor than the “desktop” here is the spreadsheet. A spreadsheet is, of course a rudimentary form of a database.

In a spreadsheet that inconvenient imposition of substrate has gone: a “document” is nothing more that an information string: more or less costless to generate, transport, replicate and store. By simply appending metadata, we can enrich it and put the same thing in several places at once. We transcend the Euclidian geometry of physical space.

Now, I said our reverence it had for the sacred substrate fell quickly away. It did not entirely fall away. We still revere wet ink, for some reason counterparts clauses, and the dear old desktop. For still, as we file, we cannot resist the siren call of folders. Folders in folders in folders in folders in folders. Why do we persist with folders?

More than twenty years ago Tom Zingale taught a young JC a valuable lesson. Battling with some byzantine folder structure, and losing, JC cried out in anguish, and Tom said this:

JC: How on earth am I meant to organise all this?

Tom: With metadata.

JC: Er, with what?

Tom: Metadata. The answer to your question is metadata. Metadata, metadata, metadata. Whatever your question is, the answer is metadata.

JC: Well, my question is, “How do I use metadata to fix this filing problem?

Tom: Oh, right. Simple: SharePoint.

About SharePoint

Now a lot of good people viscerally hate SharePoint. And, to be sure, Microsoft seems to have gone out of its way, over 20 years, to make SharePoint as hard to love as it can. But at the same time, it has based its entire Office 35 Suite on the SharePoint platform. It is, to be sure monumentally confusing, the Teams integration is baffling. The utterly dismal online versions of its Office suite drive people righteously up the wall.

But, still, a good part of the enmity for SharePoint arises from this fundamental misunderstanding. SharePoint is the first, philosophically, digitally native operating system.

SharePoint has abandoned the desktop metaphor.

SharePoint uses the spreadsheet metaphor.

In SharePoint you organise by metadata, not by folders.

DO NOT USE FOLDERS IN SHAREPOINT.

Folders are top-down. Metadata is bottom-up. Folders prefer form over substance. Metadata prefers substance over form.

SharePoint allows you to do exactly the same thing with a document library as Excel allows you to do with a spreadsheet.

So it is odd — isn’t it? We intuitively understand the power of metadata when we are presented with a spreadsheet. But the same power does not occur to us when we are presented with a file management system. The desktop metaphor is burned on our retina.

Even though it is, in essence, a supercharged online spreadsheet, SharePoint continues to be resented by almost everyone.

See also

References

  1. It was well ahead of its time: the GUI would not become mainstream until Apple released its Macintosh a decade later, in 1984.
  2. All physical information is eventually destined for the Iron Mountain.
  3. Okay: almost no form. Compared with physical information. In this section take the word “almost” as read.
  4. Grammar pedants’ corner: Even though “data” is plural, “metadata” is generally treated as a singular mass noun. Please direct your letters to the Royal Statistical Society — not because it is their fault: rather, they might keep metadata about this sort of thing.
  5. Indeed, the Windows operating system doesn’t treat folder names as metadata at all, which is mad.
  6. As in, a lookup to an object in a people directory, and not just a text name.
  7. My favourite was 001.9.
  8. I have a lengthy essay about the gradual extraction of data from the substrate but can't for the life of me find it at the moment.