End-to-end principle

The Devil’s Advocate™ Go Premium Alas, poor Yorrick, yesterday. (I knew him, you know, Horatio.) In which the curmudgeonly old sod puts the world to rights. Index — Click ᐅ to expand: The Devil’s Advocate Comments? Questions? Suggestions? Requests? Insults? We’d love to 📧 hear from you. Sign up for our newsletter.

“I once got all the way from Glasgow to Edinburgh without a ticket. I walked.”

— Sid Snot, The Kenny Everett Video Show.

“There are more network use-cases in heav’n and earth, Horatio, than are dream’t of in your philosophy.”

— Shakespeare, Spamlet, I: v

The end-to-end principle is a design framework for networks and complex systems. It says this:

Keep the network as simple as possible. Put all the complications, intelligence and complexity at its edges. Let people build whatever structures they like on it — if it is a digital commons, there will be unlimited scope for other users to build their own castles in the air, but the most basic, common, layer must be as clear and simple as it can be.

Lawrence Lessig lays out the concept very well in his magnificent Code: Version 2.0.

Overview

Network layers

You can see any network as a series of layers, with only the most basic layer connecting every client on the network. Each successive layer is more complicated, specialised, but will only be suitable for a more limited number of “clients” with specific use-cases. That bottom layer is universal — literally, since every client of the network operates on it — and must be as simple and uncomplicated as possible. Any complication in a lower-level network has costs for all clients interacting with the network through that layer, if they do not need the feature that complication provides. So the basic idea is to put complicating features in the highest possible layers such that only clients operating at that layer (or higher) needs that feature.

Conceptual example: transport network

Say we are organising transport around an area of uncultivated, flat land.

• Most basic level: The whole area, as it is, at its most basic level the network is maximally simple and flexible. You can access the “network” from anywhere (i.e., wherever you happen to be is an "entry-point" to the network) but you have to put in all the effort if you want to get anywhere. See Sid Snot quote above.
• Second layer: You can overlay a second layer suitable for people with a certain mode of transport. It will be necessarily less comprehensive than the basic layer, and will not suit all people, and will therefore have with a more limited set of entry-points. For example, a road system on the land optimizes the land for vehicular traffic, but some cost to those wanting travel some other way (and certainly to those wanting to use land for non-transport purposes) — so you don’t pave over the whole area, but built an road network, optimised for automobiles. People wanting to use the road the network need a car; people with other needs don’t use the road network at all. Once you have a car you can go anywhere on the road network; to go elsewhere in the area, you need to ditch your car and walk.
• Third layer: Say someone then operates a bus service on the road network. This is a third layer excellent for those who want to go whether the bus is going on the road network, but no good if you want to go somewhere else on the road network, let alone if you want to go elsewhere on the land.

The classic example is of course the internet itself. At its most basic layer, it is a series of connected notes using the TCI/IP protocol and packet switching to which takes any type of data (text, video, audio etc) breaks it into tiny homogenous “packets” and sends them off across the network with an address, and the client at the receiving end reassembles them. There are all kinds of additional layers in it, be they walled gardens (Facebook etc) streaming services (Netflix), cloud servers and what not. Even as the fortunes of these services and gardens wax and wane (where are they now, AOL and MySpace) the un-owned internet grows ever stronger.

Design

Systems theory

We don’t want to sell you Life Insurance . . we want you to know and have what life insurance will do. A 1/4 million drills were sold last year: no one wants a drills and holes. What they want is the hole.

—The Manhattan Mutual Life Company advertisement, Manhattan Kansas, 1946

A new tool is not an end-goal in itself but a means to improving an existing system to create a certain output. In figuring out how to improve that system bear in mind the stocks, flows and feedback loops already in that system: these are what you need to influence; your machine needs to influence them. So bear in mind:

Systems develop for reasons that are beyond the control of the system owner: indeed, most systems are not owned. Not all the reasons motivating the configuration of the system will be good ones from the perspective of an end user (in any agency scenario the hard to persuade an entrenched agent that she her role in the revised system should change — she has invested her capital in getting to where she is, and will not want to give up that capital, nor control over that position. If her role in the revised system will be deprecated — will change for the worse — it will be impossible to persuade her to adopt the new system. Why should she?

But no participant in the system has a monopoly on it. If the system appears to necessitate such a behaviour change: to oblige someone to embark on a new journey that they don’t control, the better approach might be to leave that agent in situ, as it, but give other participants in the system an alternative route to achieving their goal. In the digital commons this is particularly easy to do. This way you have changed the dynamics of the remainder of the system in a way that obliges the agent in question to change her approach.

Take the arrival of email. Had its success criteria involved shutting down the physical email system first, and replacing it with email — obliging people to buy the computer equipment and networks needed before a single switch-over day, it would never happen.

Do this

This design principle has a number of advantages, especially in the digital commons where issues of physical property and the tragedy of real commons don’t apply:

• It’s open source: no-one needs to sanction the creation of a new layer. You can innovate on the network — create your own new bus company — without bothering the network “owner” 9if there even is one — there isn’t, on the internet. This also avoids “strategic” behaviour by the network owner (rent-seeking, monopoly behaviour).
• It’s permissive: no-one is forced to use a more complex layer — clients retain autonomy and flexibility to select optimal levels at which to engage with the network: you can decide to walk, take a bus, take a car etc).
• It’s competitive: For the same reason, it incentives efficient and effective design: one can drop down to a more basic layer and instead implement an alternative, competing layer.
• It’s agile: Thus as use cases develop, the network develops organically to accommodate the new uses through the entrepreneurial development of the network. So the internet has adapted from messaging to webpages, to VOIP and video-streaming etc etc etc.
• It’s iterative: It works well in an complex scenario because it encourages maximum iteration: provisional structures which can be freely adapted by any participant on the network to adapt to the changing environment in which the network.

Don’t do this:

This applies in the design of all other kinds of multi-use systems, such as — for rather good example — a digital execution hub. There are a number of things you should not do when designing a network if you want to maximise its chances of success:

• Teach to fish, do not give a fish: Do not try to anticipate future uses. Acknowledge that the potential applications and use-cases
• Set it free: Release all code. No copyright. Make access free. If you try to meter it, extract rent or otherwise monetise basic access to the network, or stifle user attempts to build structures on the network, you will dissuade people from using the network at all.
• Live with free-riders: it is better to permit free riders than to charge entry to all. Free riders aren’t such a bad thing: mostly, they are not exclusively free-riders and their usage patters will give other users valuable data about what parts of the network are useful and which are not. free-riders who contribute nothing are using your code, and that means you get to influence community standards. In any case, the costs of policing free-riders, and adverse impact that would have on the network’s success (metering, pay-walling, charging) outweighs the benefit to everyone (yes, including the free-rider) of a free, open, adaptive network.
• Bottom up, not top down: Don’t design by committee. As far as possible allow — and require — clients to develop structures on the network that are useful to them. The premise is that the basic layer of the network is a simple as possible: once established as a design it should not need maintenance, and users should contribute much of the hardware.

See also

• Code: Version 2.0
• Iteration
• Doubt