End-to-end principle

The design of organisations and products →Go uptown Alas, poor Yorrick, yesterday. (I knew him, you know, Horatio.) Index: Click ᐅ to expand: Design Tell me more Sign up for our newsletter — or just get in touch: for ½ a weekly 🍺 you get to consult JC. Ask about it here.

“I once got all the way from Glasgow to Edinburgh without a ticket. I walked.”

— Sid Snot, The Kenny Everett Video Show.

“There are more network use-cases in heav’n and earth, Horatio, than are dream’t of in your philosophy.”

— Shakespeare, Spamlet, I: v

The end-to-end principle is a design framework for networks and complex systems. It says this:

Keep the network as simple as possible. Put all the complications, intelligence and complexity at its edges. Let people build whatever structures they like on it — if it is a digital commons, there will be unlimited scope for other users to build their own castles in the air, but the most basic, common, layer must be as clear and simple as it can be.

Lawrence Lessig lays out the concept very well in his magnificent Code: Version 2.0.

This minimalism in the Internet’s design was not an accident. It reflects a decision about how best to design a network to perform a wide range over very different functions. Rather than build into this network a complex set of functionality thought to be needed by every single application, this network philosophy pushes complexity to the edge of the network—to the applications that run on the network, rather than the network’s core. The core is kept as simple as possible. Thus if authentication about who is using the network is necessary, that functionality should be performed by an application connected to the network, not by the network itself. Or if content needs to be encrypted, that functionality should be performed by an application connected to the network, not by the network itself.

This design principle was named by network architects Jerome Saltzer, David Clark, and David Reed as the end-to-end principle. It has been a core principle of the Internet’s architecture, and, in my view, one of the most important reasons that the Internet produced the innovation and growth that it has enjoyed.

Later

As I’ve already described, the Internet embodied this principle by keeping the functionality of TCP/IP focused quite narrowly—that is, on the single function best-efforts delivery of packets of data. What those packets do, or who they’re meant for, is not a concern of the protocol. Just delivering packets is the end.

One consequence of this design, then, is that people can innovate for this network without any need to coordinate with any network owner. If you want to develop an application to deliver voice across IP, then all you need to do is to write the application to use the TCP/IP protocols to send data across the network in a way that will make your application run.

This design embeds a value that encourages innovation in applications for the network. It does so both because it minimizes the costs of developing new applications (you don’t need the hassle of asking or clearing permission with anyone) and because it avoids strategic behavior by the network owner.

Consider again the idea of developing a Voice-over-IP application. If the network is owned by the telephone companies, they would not be excited about an application that will cannibalize their telephone market. Thus, if permission were required before the VOIP application could be deployed, we might well expect the VOIP application not to be deployed—either because someone developed it, but it was blocked, or because smart developers knew it was a waste of time to develop it, because it would be blocked. As Susan Crawford describes, “The miraculous growth of the Internet has in large part come from the nondiscrimination against higher levels. . . . Innovators at the application layer have been able to assume the continued stable existence of the lower layers.”

The value here is innovation and competition. The network empowers the widest range of innovators—users of the network—and entitles all of them to innovate for this network. Any innovation can be deployed on the network (so long as it respects the TCP/IP protocols). If users of the network like the innovation, then the innovation is a success.

Simultaneously—at least so long as the e2e principle is respected—this design disables the potentially most powerful actor in the network, the network owner, from interfering with the opportunity for innovation within the network. The network owner might not like the stuff being developed, but e2e disables the opportunity to block that development.

Overview

Network layers

You can see any network as a series of layers, with only the most basic layer connecting every client on the network. Each successive layer is more complicated, specialised, but will only be suitable for a more limited number of “clients” with specific use-cases. That bottom layer is universal — literally, since every client of the network operates on it — and must be as simple and uncomplicated as possible. Any complication in a lower-level network has costs for all clients interacting with the network through that layer, if they do not need the feature that complication provides. So the basic idea is to put complicating features in the highest possible layers such that only clients operating at that layer (or higher) needs that feature.

Conceptual example: transport network

Say we are organising transport around an area of uncultivated, flat land.

Most basic level

The whole area, as it is, at its unimproved, is the basic level the network. It is maximally simple and flexible. You can access the “network” directly, by walking, and from anywhere (i.e., wherever you happen to be is an "entry-point" to the network) but you have to put in all the effort if you want to get anywhere. You can access the network through a “device” you supply yourself — a vehicle — but not all vehicles work well directly on the basic network. Tractors and tanks do, but other vehicle types might go faster if there were a second layer on part of the network.

Second layer

Such a second layer — a road network, for example — enable those with certain transport “devices” to move very efficiently from point to point on that second network layer. It will be necessarily less comprehensive than the basic layer, will have with a more limited set of entry-points and will not suit all “clients” of the network. Roads are optimised for vehicular traffic, but those having to travel some other way (and those wanting the land for some other purpose) will find the road has a cost. Those wanting to use the road the network need to bring an expensive and highly complicated device with them: a car. You are not permitted to access the network without one, and version vehicles are practically, or legally, prohibited — trains, combine harvesters, planes. There are some who wish too use the road network, but don’t have their own device. For these people, third layers evolve. Bus services, taxi services, ride-sharing services.

Third layer

Third layer networks bring access to the network to those without their own device, but at a cost of flexibility. The bus only goes at certain times, to certain places. Taxis and ride shares have an elevated running cost and are somewhat less convenient for regular use than private vehicles.

T’internet

The classic case of end-to-end design is of course the internet itself. At its most basic layer, it is a network of interconnected nodes. You send any data type (text, video, audio etc) across it using a routing protocol (the TCI/IP protocol) which decomposes the file into tiny “data packets”, addresses them, and routes them individually across the network. Each packet goes its own way, through the network, and is only reassembled into a file by the recieving client at the other end of the network. There are all kinds of additional layers in it, be they walled gardens (Facebook etc) streaming services (Netflix), cloud servers and what not. Even as the fortunes of these services and gardens wax and wane (where are they now, AOL and MySpace) the un-owned internet grows ever stronger.

Design

Systems theory

We don’t want to sell you Life Insurance . . we want you to know and have what life insurance will do. A 1/4 million drills were sold last year: no one wants a drill. What they want is the hole.

—The Manhattan Mutual Life Company advertisement, Manhattan Kansas, 1946

“I don’t think it works like that at all. You see an electric drill in a shop and decide you want it. Then you take it home and wander around your house looking for excuses to drill holes in things.”

—Llewelyn Thomas, quoted in Rory Sutherland’s Alchemy

A new tool is not an end-goal in itself but a means to improving an existing system to create a certain output. In figuring out how to improve that system bear in mind the stocks, flows and feedback loops already in that system: these are what you need to influence; your machine needs to influence them.

Networks are systems. They develop, Ouija board-style, as a result of their actual use. Development is beyond the control of the system owner: indeed, most systems are not owned.

Not all the reasons motivating the configuration of the system will be good ones from the perspective of an end user (in any agency scenario, it is hard to persuade an entrenched agent that she her role in the system should change — she has invested her capital in getting to where she is, and will not want to give up her position, nor control over how she develops it. If her role in the revised system will be deprecated — will change for the worse — it will be impossible to persuade her to adopt the new system. Why should she? Hence the eternal dilemma of the change manager: you can take a horse on a customer journey to water, but you can’t make her drink.

But no system participant has a monopoly: that's a fundamental network design principle. If the system has an open architecture — conforms to the end-to-end principle — then be entrepreneurial: build your new layer in the system and see if the system adjusts to it. Leave the recalcitrant agent in situ, but give other system participants an alternative route to their goal. In the digital commons this is easy to do. This way you have changed the dynamics for the remainder of the system in a way that obliges the recalcitrant agent to change her approach. If she does not, water will flow around her, like traffic around a pedestrian at a neopolitan roundabout.

Take the arrival of email. Had its success criteria involved “taking the physical postal system on a journey” to convert it to email — obliging people to buy the computer equipment and networks needed before a single switch-over day, it would never have happened. How would you even start on that process?

Do this

This design principle has a number of advantages, especially in the digital commons where issues of physical property and the tragedy of real commons don’t apply:

• It’s open source: no-one needs to sanction the creation of a new layer. You can innovate on the network — create your own new bus company — without bothering the network “owner” 9if there even is one — there isn’t, on the internet. This also avoids “strategic” behaviour by the network owner (rent-seeking, monopoly behaviour).
• It’s permissive: no-one is forced to use a more complex layer — clients retain autonomy and flexibility to select optimal levels at which to engage with the network: you can decide to walk, take a bus, take a car etc).
• It’s competitive: For the same reason, it incentives efficient and effective design: one can drop down to a more basic layer and instead implement an alternative, competing layer.
• It’s agile: Thus as use cases develop, the network develops organically to accommodate the new uses through the entrepreneurial development of the network. So the internet has adapted from messaging to webpages, to VOIP and video-streaming etc etc etc.
• It’s iterative: It works well in an complex scenario because it encourages maximum iteration: provisional structures which can be freely adapted by any participant on the network to adapt to the changing environment in which the network.

Don’t do this:

This applies in the design of all other kinds of multi-use systems, such as — for rather good example — a digital execution hub. There are a number of things you should not do when designing a network if you want to maximise its chances of success:

• Teach to fish, do not give a fish: Do not try to anticipate future uses. Acknowledge that the potential applications and use-cases
• Set it free: Release all code. No copyright. Make access free. If you try to meter it, extract rent or otherwise monetise basic access to the network, or stifle user attempts to build structures on the network, you will dissuade people from using the network at all.
• Live with free-riders: it is better to permit free riders than to charge entry to all. Free riders aren’t such a bad thing: mostly, they are not exclusively free-riders and their usage patters will give other users valuable data about what parts of the network are useful and which are not. free-riders who contribute nothing are using your code, and that means you get to influence community standards. In any case, the costs of policing free-riders, and adverse impact that would have on the network’s success (metering, pay-walling, charging) outweighs the benefit to everyone (yes, including the free-rider) of a free, open, adaptive network.
• Bottom up, not top down: Don’t design by committee. As far as possible allow — and require — clients to develop structures on the network that are useful to them. The premise is that the basic layer of the network is a simple as possible: once established as a design it should not need maintenance, and users should contribute much of the hardware.

See also

• Code: Version 2.0
• Iteration
• Doubt