Say what you want, not what to do: desired state and the reconcile loop

§ 01 Year 3: the same 3 a.m.

It’s now Year 3, and Fennel runs on managed Kubernetes from its cloud provider. The provider runs Kubernetes’ own machinery, while Fennel’s services run on four machines, numbered 01 to 04 as before. Kubernetes calls each machine in a cluster a node. Ari, the platform lead Fennel hired this year, led the move.

One Saturday at 03:04, machine 03 dies again. Kubernetes knows it as node 03 now, and just as a year ago, it was running orders and a dispatch copy. This time nobody’s phone rings. Sam sleeps through it and reads about it over breakfast.

On Monday, Jordan asks Ari how Kubernetes knew what to do.

“Nobody told it what to do. We told it what we want.”

Ari

The rest of this chapter explains what Ari means, and what that sentence doesn’t promise.

§ 02 Say what you want, not what to do

When a machine dies, there are two ways to write down how Fennel should recover. Fig. 01 compares both for the same incident. On the left is the step-by-step task list Sam worked through after machine 03 died last year. On the right is a goal that says how many copies of each service must keep running.

Sam’s list spells out what to do, step by step, and engineers call that style imperative. Fennel’s goal describes what should be running and leaves the steps to the system, which makes it declarative. What a declarative goal describes is called the cluster’s desired state.

The difference matters most on a night nobody planned for. A task list is only right for the failure its author imagined. A goal doesn’t mention machines, so it stays true whichever one fails. It still shows what’s missing, and whatever acts on the goal can work out the steps for itself.

Fennel writes its goals in files called manifests. Fig. 02 shows a shortened version of the manifest for storefront. Its line replicas: 3 is the desired state: three copies. Teams usually keep manifests in version control, so that a change to a goal can be reviewed before it goes in, and leaves a history behind.

Chapter 3 ended on a question: what if the record were the thing the machines act on, instead of a note about them? With Kubernetes, that’s how it works. The cluster keeps its own record of what’s wanted and what’s running, and every part of Kubernetes works from that record.

§ 03 Pods: what Kubernetes counts

There’s one more word to learn before Ari applies the storefront manifest. Kubernetes doesn’t count or place containers directly. Instead, it wraps each copy’s container in a Pod, and Pods are what Kubernetes counts, places and replaces. Most Pods hold just one container, which is why people often use “Pod” and “container” interchangeably in conversation. Sometimes a small helper container runs alongside it in the same Pod.

§ 04 Moving day: what “apply” does

Fennel didn’t move to Kubernetes all at once. Ari’s team moved its services onto the cluster one at a time, and by the morning of moving day, orders, payments and dispatch were already running there. Moving day is the day the last service, storefront, moves too. Ari has written its manifest, the one in Fig. 02, and all that’s left is to hand it to Kubernetes.

Two parts of Kubernetes matter for what happens next. The first is the cluster’s record. It holds every goal Fennel has declared, such as “orders: 1 copy”, together with what Kubernetes knows is actually there: which Pods exist, and which node each one is on. The second is the API server, the program through which everything reads and changes that record. When Ari applies a manifest, the command sends it to the API server. The loops read the record through the API server and write their results back the same way. And on every node, a small program called the node agent regularly reports to the API server what is running on its node, so the record stays up to date.

Kubernetes was already busy before Ari applied anything. Several loops, each a small program that looks after one kind of thing, run all the time, and each one repeats the same three steps: it observes part of the cluster, compares what it sees with what the record says is wanted, and acts if the two differ. On the morning of moving day, orders, payments and dispatch were running exactly as declared, so every comparison came out even and the loops had nothing to do. Lane A shows what changes when storefront arrives.

Every loop in lane A followed the three steps described at the start of this section: observe, compare with what’s wanted, act to close any gap, and then start again. That pattern is called a reconcile loop, and it isn’t special to moving day, because almost everything Kubernetes does is built from it. A controller is one implementation of the pattern: a part of Kubernetes that runs a reconcile loop for one kind of thing. The ReplicaSet loop, for example, is the controller for ReplicaSets. Engineers call one pass of the loop “reconciling”. The scheduler and the node agent work the same way, though engineers don’t usually call them controllers.

The pattern is the same for every loop, but each loop fills it in differently, and filling it in is the quickest way to understand any one of them. Ask four questions: which loop is it, what does it want, what does it see, and what does it do about a gap? Together, these questions make up the loop lens. At the fourth step of lane A, for example, the loop was the scheduler. It wants every Pod to have a node; it saw three Pods with none; and it wrote a node into each one. Each new part of Kubernetes in the chapters ahead is another loop, so the loop lens will explain those too.

§ 05 Saturday, 03:04, again

Chapter 1 left Jordan with a question to ask about any machine Fennel depends on:

“If this box vanished tonight, how long until we’re back, and who has to be awake?”

Chapter 1

In Year 3, the answer is roughly six or seven minutes with default settings, sometimes less, and nobody has to be awake.

Those minutes still cost something. Fennel runs orders as a single copy, so it was down the whole time: when its one Pod stopped, there was no other copy to take orders while the loops replaced it. A service running as one copy is what Chapter 1 called a single point of failure. The loops restore what’s been declared, but they can’t prevent the gap while they do it. That makes this a question worth asking Ari: “Which of our services run as a single copy?”

Fig. 04 looks at each loop you’ve watched through the loop lens, and shows which of Sam’s jobs it took over.

No part of Kubernetes made a plan for 03:04. The node loop didn’t know what a ReplicaSet was, the ReplicaSet loop didn’t know why its Pods had gone, and the scheduler didn’t know there had been a failure. Each one read the record, did its one job and wrote the result back, and the next loop picked up the work from there. That’s how the work gets split up without a single conductor.

Chapter 3 called Kubernetes an orchestrator because of the jobs it does, and that’s still true. Kubernetes’ own documentation, though, warns about what the word suggests. It describes orchestration as following a fixed script, “first do A, then B, then C”, and says that Kubernetes instead “comprises a set of independent, composable control processes”. It adds: “Centralized control is also not required.” So Chapter 3 used the word for what gets done, while the documentation is talking about how it gets done.

There’s a practical reason for building it this way. The same documentation prefers “simple controllers rather than one, monolithic set of control loops”, because controllers can fail. When the loops are separate, one of them can fail without stopping the rest.

Nothing at 03:04 needed a release, or anyone applying anything, because the loops had been running all along.

§ 06 Replaced, not repaired

Ari’s dashboard shows the difference. On Friday it listed orders-5c8x on node 03, and since 03:10 it has listed orders-q2rw on node 04. Kubernetes never moves a Pod. It replaces it with a new one, made from the same template, which has a new name and a new address and keeps nothing from inside the old one.

This is why Kubernetes treats Pods as disposable. People rarely create Pods by hand; a controller makes them, and each one lasts until it finishes its work, is removed, or loses its node. The new address is also the reason storefront needed another loop, Chapter 7’s, to find the new orders Pod.

What “disposable” asks of the app

Engineers will recognise this idea from the twelve-factor app, a widely used checklist for building software, which calls it “disposability”: an app’s processes “can be started or stopped at a moment’s notice”. For a service running on Kubernetes, that comes down to four things:

• Keep nothing important inside the Pod. Fennel’s orders live in its database, outside the app, as Chapter 2 explained; where that database should run is the subject of Chapter 10.
• Cope with being stopped or replaced at any moment.
• Start in seconds rather than minutes, because a slow start lengthens the gap every time a Pod is replaced.
• Finish any work in progress when asked to stop. By default, Kubernetes gives a Pod 30 seconds to do that.

Kubernetes can’t do these for you; they’re up to the app. They’re the questions to ask whenever one of your services moves onto the cluster.

Self-healing, honestly

You’ll hear what happened at 03:04 called self-healing. After last year’s 3 a.m. pages, it’s tempting to hear that as “Kubernetes makes our app reliable”. What the loops actually do is restore the declared state: they restart containers that crash, and they replace the Pods of nodes that die. There are several things they can’t do:

• Fix a bug. An app that crashes as it starts is restarted, crashes again, and is restarted again, forever. §08 shows an example.
• Find room that isn’t there. If no node has room for a new Pod, the Pod waits, unplaced, until room appears. Chapter 5 gives this state a name.
• Bring back what was inside a Pod. Anything written inside a Pod is lost with it; keeping data safe is the subject of Chapter 10.
• Keep a single copy up. As at 03:04, a service with only one copy is down while that copy is replaced.

In other words, Kubernetes restores the number of copies that are running. Whether the app itself is healthy is still your engineers’ responsibility.

§ 07 Who runs what

You’ve now met every part of a cluster that a manager needs to know by name. Fig. 05 brings them together and shows who runs each part.

You met the API server in §04. Everything that reads or writes the record goes through it, whether that’s a person applying a manifest, one of the loops, or a node agent reporting in. The node agent on each node is what engineers call the kubelet; it starts whatever has been assigned to its node and reports back. Together, the API server, the record behind it, and the loops that work on that record, apart from the node agents, make up the control plane.

On a managed service, the provider runs the whole control plane in every mode, and carries out its upgrades to new Kubernetes versions. Deciding when to upgrade, and checking that Fennel’s apps still work on the new version, is still partly Fennel’s job; that work is the subject of Chapter 14.

Who looks after the nodes depends on the mode. In each big provider’s standard mode, the customer does, which at Fennel means Ari’s team. In the fully managed modes, the provider does. Either way, Fennel pays for the computing its Pods run on, which in most modes means paying for the machines themselves. Some things are always Fennel’s responsibility: what goes into the record, its images and code, its data, who may change what, and watching and responding to alerts.

Without a managed service, Fennel’s own team would also have to run the control plane, including keeping the record safe.

§ 08 The conversation

Fennel’s newest service, notifications, sends customers texts and emails about their orders. If it runs late for a while, that’s annoying but not a disaster. On Wednesday, a setting it needs goes missing.

On engineers’ dashboards, this state shows up as CrashLoopBackOff. An example in the Kubernetes documentation shows a Pod in this state that has restarted 200 times over two days and nine hours.