§ 01 Year 3: releases without a release night
Cast
- Jordan
- Co-founder, managing the engineers: that’s you, in this story.
- Ari
- Platform lead. Runs Fennel’s cluster.
- Riley
- An engineer on the
catalogteam. - Morgan
- Co-founder. Runs Fennel’s money.
- Sam
- The founding engineer, who used to run every release by hand.
I didn’t read the last chapter. What happened?
- In Chapter 4, Fennel moved onto managed Kubernetes. The cluster keeps a record of what Fennel wants running, and many small loops each compare one part of that record with what is actually running, and act to close any gap. The ReplicaSet loop keeps a set number of Pods (Kubernetes’ wrapper around a running container), the scheduler gives each new Pod a node (a machine in the cluster), and the node agent on that node starts it, and restarts it if it crashes.
- In Chapter 5, Morgan asked why the first cloud bill after the move had gone up. The answer lay in two numbers each team writes down for its Pods: what each Pod reserves, its requests, in GB of memory and processor cores, which the scheduler places Pods by; and the most it may use, its limits. On a Friday at 17:45, a
dispatchPod waited, pending, because no node had room for it, until Ari’s team added two more nodes. The cluster has had five nodes since: 01, 02, 04, 05 and 06. - To understand any loop, ask four things: which loop is it, what does it want, what does it see, and what does it do about a gap?
In Chapter 3, Sam updated storefront from v12 to v13 by hand: start one v13 copy, send customers to it, stop one v12 copy, three times over, while nobody else shipped anything. On Kubernetes, what does a team write down to release a new version?
In Fennel’s first years, a release was an event. Every team’s changes waited for a release night, when Sam put the new versions out by hand, one copy at a time, after posting in the team chat: “Nobody deploy anything, I’m updating storefront.” The update you may remember from Chapter 3, storefront from v12 to v13 on a Tuesday, went the same way. It took Sam the evening, and until it was done nobody else could ship anything.
Year 3 looks different. A few weeks after Morgan’s question about the bill, the team that looks after product listings and prices moved them out of storefront into a service of its own, catalog, so that it could change prices and listings without waiting for the website’s releases. Riley’s team now ships catalog two or three times a day. When one of their changes is merged, Fennel’s build pipeline builds a new image and hands it to the cluster, and the release happens while everyone keeps working. Nobody posts in the chat any more.
This chapter follows one Thursday of those releases, and it starts with a question Jordan put to Ari a few days before.
Jordan“So who’s doing what Sam did on release nights?”
Ari“A loop. We tell it which version we want.”
Jordan“And if the version is bad?”
Ari“That depends on how it’s bad.”
§ 02 One record per service: the Deployment
In Chapter 4, the manifest Ari applied for storefront described a ReplicaSet: a record that says “keep this many identical copies of this Pod”, together with the image each copy runs. The ReplicaSet loop then keeps that many Pods running and replaces any that disappear. That is all a ReplicaSet does: it keeps a number of copies of one Pod description, and engineers rarely write one themselves. What they write instead looks almost the same, and Fig. 01 shows a shortened version of catalog’s.
kind: Deploymentthe kind of record: a version, and how many copies
metadata: name: catalogspec:Chapter 4’s ReplicaSet held this part
replicas: 3as in Chapter 4: three copies
template:what each copy is; changing it starts a rollout
# … labels, checks and # other details left out spec: containers: - name: catalog image: catalog:v62the version to run; the pipeline changes this line
The first line names the kind of record. This one is a Deployment, and it’s what Fennel’s teams write for each of their services.
New term · 01 / 04
Deployment
A record of which version of a service should run, and how many copies. Its loop gets there by keeping one ReplicaSet per version and resizing them.
What engineers write instead of Chapter 4’s ReplicaSet.
A Deployment doesn’t run Pods itself. For the version it currently wants, it keeps a ReplicaSet, and Chapter 4’s ReplicaSet loop keeps that ReplicaSet’s Pods running. When someone changes the Pod description, which usually means the image line, the Deployment loop makes a new ReplicaSet for the new version and then, gradually, makes the new ReplicaSet bigger and the old one smaller. Changing only the number of copies doesn’t start any of this, because every copy is still the same version; changing what each copy is does. So engineers write and change the Deployment, and leave its ReplicaSets to it, as its machinery.
In Chapter 4’s terms, the Deployment loop is one more implementation of the same reconcile pattern, and you can fill in its loop lens. It wants the version and the number of copies that its record names. It sees its ReplicaSets, one per version, and how many of their Pods are ready. What it does about a gap is move copies from the old version’s ReplicaSet to the new one, a few at a time. What “a few at a time” means, and what “ready” means, are what the next three sections are about.
Where does a new version come from? When Riley’s team merges a change, Fennel’s build pipeline, which engineers call CI, builds a new image, catalog:v63, and pushes it to Fennel’s registry, where images are kept and, as Chapter 2 showed, never change once they are built. Then the pipeline does one more thing: it writes the new tag into the image line of catalog’s Deployment. That one edit to the record is the whole release, and everything that happens after it is the loops’ work. (Pipelines differ from team to team; this is how Fennel’s works.)
§ 03 Thursday, 10:15: a routine release
When catalog arrived in the cluster, the scheduler put its three Pods where there was room, which by then meant the two newest nodes: one on node 05, next to the dispatch Pod that waited for room on that Friday at 17:45 in Chapter 5, and two on node 06. Each catalog Pod reserves 2 GB of memory. The other three nodes, 01, 02 and 04, are nearly full and nothing on them changes this morning, so the drawing leaves them out. At 10:15 on Thursday, the pipeline writes catalog:v63 into the Deployment’s record.
W-01 follows the next minute step by step as you scroll. The drawing is Chapter 5’s, with three changes. The record now has a Deployment row, showing the version it wants, above one row for each of its ReplicaSets. Each catalog Pod carries its version in its label and in its colour: v62 in magenta, v63 in blue. And Pods say READY rather than RUNNING, because from now on that is the difference that matters: Kubernetes sends a Pod customers’ requests only once it is ready, which means once it passes a check that the team wrote. §05 opens that check up.
-
PREDICTCheck 02 / 07
The v63 Pod is running on node 05 but isn’t ready yet, and
catalognow has four Pods where the record wants three. What does the Deployment loop do next? -
Answer the question above to see what happens
-
Answer the question above to see what happens
-
Answer the question above to see what happens
W-01Fennel’s nodes, versions and timings are fiction. With three copies and the default settings, the Deployment loop ran one extra Pod at most and kept three ready throughout.
What the Deployment loop did between 10:15 and about 10:16 has a name: a rolling update.
New term · 02 / 04
rolling update
Replacing a service’s Pods with a new version a few at a time, keeping enough Pods ready throughout: old ones go only as new ones become ready.
Sam’s Tuesday, done by a loop.
If those rounds looked familiar, they should. In Chapter 3, Sam updated storefront from v12 to v13 in the same three rounds of start one, stop one, using one copy’s worth of spare room. What changed is everything around the rounds. The loop took about a minute rather than an evening. Nobody had to post “Nobody deploy anything”, so the other teams could go on shipping while it ran. And it was the team’s check, rather than Sam looking, that decided when each new copy was ready. One detail is easy to miss: the emptied v62 ReplicaSet is still in the record, and §07 shows why that matters.
§ 04 Room to ship
How many Pods the Deployment loop moves at a time is set by two numbers in the Deployment. With default settings, a rolling update may run up to a quarter more Pods than the record wants, rounded up, and may let up to a quarter of the wanted Pods be unavailable, rounded down. For catalog’s three copies, a quarter is three-quarters of a Pod, which rounds up to one extra Pod and down to none short. So at every moment of the rollout at least three Pods are ready to serve customers, and the loop never adds more than one extra Pod. Teams can change both numbers; that’s for your engineers to tune.
The extra Pod has a cost. While it runs, it needs room on some node, which for catalog means 2 GB of memory and a quarter of a processor core. In Chapter 5, room kept free on purpose looked like insurance against losing a node. It turns out to be room to ship as well.
Suppose the cluster were as full as on that Friday at 17:45 in Chapter 5, when a dispatch Pod waited for room: no node has 2 GB free. With the default settings, what happens when the pipeline writes catalog:v63 into the record?
In that case nothing breaks, and nothing rolls back. The new Pod waits, pending, as the dispatch Pod did on that Friday, and the scheduler keeps trying to place it. Meanwhile the three v62 Pods go on serving customers, and the release simply doesn’t happen until someone makes room. So releases, too, depend on the spare room Chapter 5 was about, and that gives Jordan a question for the list.
“Do we keep enough spare room to release during the busiest hour?”
The question to ask
What about a service with more copies?
With four copies, a quarter is exactly one Pod, so the default settings allow one extra Pod and one short. The Deployment loop may then remove an old Pod before any new one is ready, and it moves roughly two at a time, running at three of four ready copies for a while. That means a larger service can make progress even when there’s no spare room at all, by giving up a copy for a while. Teams can set both numbers to suit each service, including allowing no extra Pod at all.
§ 05 What “ready” means
How did the Deployment loop know that the first v63 Pod was ready? It didn’t look at the Pod; it read a result. The node agent on each node, Chapter 4’s loop that starts and restarts containers, also runs checks that the team wrote into the Deployment’s description of each Pod. It runs them again and again, every ten seconds by default, and reports each result. Engineers call these checks probes, and a Pod can have two kinds, which ask different questions. Fig. 02 puts them side by side.
Runs both, every 10 s by default
Readiness probe
“Ready for traffic?”
readinessProbe:
httpGet:
path: /healthzthe app is up and can reach its database
any success code (200–399)
keeps running; sent no requests; a rollout doesn’t count it
always passes: ready once running
Liveness probe
“Still alive?”
livenessProbe:
httpGet:
path: /livezthe app is up and not stuck
any success code (200–399)
the node agent restarts the container
always passes
The left-hand check is the one the rollout relied on. It is called a readiness probe.
New term · 03 / 04
readiness probe
A check, written by the team and run by the node agent throughout a Pod’s life, that decides whether the Pod may be sent traffic.
Ready for traffic. Not “correct”.
catalog’s readiness probe asks its question by requesting a path, /healthz, from the app, and any success code counts as a pass. While the probe fails, the Pod keeps running, but it is sent no customer requests (that’s the work of a loop we’ll name in Chapter 7), and a rollout doesn’t count it as ready. The probe goes on running for the Pod’s whole life, not only when the Pod starts, so a Pod that stops being ready later is taken out of traffic, and put back once it passes again.
The right-hand check is the liveness probe, and it asks a different question: is the container still alive? When it fails, the node agent restarts the container, just as it restarted the crash-looping notifications after each crash in Chapter 4. It’s there for a job that Chapter 3 left for later, noticing a container that is running but stuck: one that has stopped making progress, for example because it is waiting for something that will never happen, which no crash would ever reveal.
And if a team writes no probe at all? Then Kubernetes treats the missing check as passing. Without a readiness probe, a Pod counts as ready as soon as its container is running.
During Friday’s rush, one catalog Pod loses its connection to the database for a minute. catalog’s readiness check asks whether the app can reach its database; its liveness check asks only whether the app is up and not stuck. What happens to that Pod?
§ 06 Two ways to be bad
What you watched at 10:15 is a real safety net. At no moment were fewer than three catalog Pods ready, and each v63 Pod took over from a v62 Pod only after its check had passed. So it’s tempting to conclude that a release which finishes rolling out must be a good release, and engineers say as much when they report that a new version “passed its health checks”. Before you read on, test that conclusion on the afternoon’s release, v64.
Two ways v64 could be bad. In the first, it crashes as soon as it starts, because of a missing setting (as notifications did in Chapter 4). In the second, it starts, connects to its database and serves pages normally, but every price shows as $0.00. With the default settings, what does the rollout do with each?
Answer the question above to see the two outcomes
If v64 crashed on start · 14:03
v64 as shipped · 14:03
The right-hand panel is what happened on Thursday at 14:02. About a minute after the pipeline wrote catalog:v64 into the record, every catalog Pod was running v64, and every customer was seeing prices of $0.00.
Why did the rollout treat the two versions so differently? A container that crashes as soon as it starts never stays up long enough to be ready, so the Deployment loop never has a ready v64 Pod to count, never removes a v63 Pod, and the rollout stops where it is. The $0 version, on the other hand, started normally and answered its check with a success code, so it was ready by every measure Kubernetes has. A check passes on the response’s status code alone, and nothing in Kubernetes looks at what the app sends back. catalog’s check asks whether the app is up and can reach its database; it never looks at a price. A rolling update protects availability, which means there are always enough copies running to serve customers. It knows nothing about correctness, whether those copies are doing the right thing.
Notice, too, what “stops” means in the left-hand panel. The rollout stops advancing: the v64 Pod keeps crash-looping, the v63 Pods keep serving, and Kubernetes waits for someone to roll back or ship a fix. It doesn’t go back to v63 by itself.
How long does Kubernetes wait on a stuck rollout?
By default, if a rollout makes no progress for 10 minutes, Kubernetes marks it as having failed to progress, and keeps retrying. That’s all it does. A pipeline or an add-on tool can watch for that mark and roll back automatically, if a team sets one up.
A rollout gets stuck only when the new Pods never become ready. A version that runs for a few seconds before it crashes can briefly count as ready, and the rollout can move on past it. That is one reason teams write a readiness check, and they can also ask for a new Pod to stay ready for a while before it counts.
ANALOGYYou know this from shift changes. A shop manager sends the morning cashier home only once the afternoon cashier has signed in and opened a till, so that the shop is never left without someone serving. Here, the Deployment loop works the same way: it removes an old Pod only once a new one has passed its check.
WHERE THIS ANALOGY BREAKSA manager at the counter would notice a new cashier charging nothing for the groceries within a minute. The Deployment loop never sees a single sale. It sees only the result of the check the team wrote, and as long as that passes, it carries on.
To catch wrong behaviour before every customer sees it, teams add a step that a plain rolling update doesn’t have. In a canary release, the new version goes to a small share of customers first, while someone, or something, watches whether it behaves correctly, and only then replaces the rest. In a blue-green release, the new version runs beside the old one and is tested there, before all traffic is switched over to it at once. A plain Deployment does neither by itself, and it doesn’t roll back on its own either; tools such as Argo Rollouts add these on top of Kubernetes. Fennel uses neither, yet.
§ 07 Thursday, 14:02: going back
The afternoon’s release, v64, changed how catalog reads prices from its database. One field in the database had been renamed, so every price came back empty, and the app showed an empty price as $0.00. The change passed the team’s tests, the pipeline built catalog:v64, and at 14:02 it wrote the new tag into the Deployment. Fig. 04 shows the six minutes that followed.
Look at who noticed. Not Kubernetes, which had done exactly what it was built to do, and not a dashboard: it was Morgan, about three minutes in, from the orders. (How a team notices this kind of problem sooner is a subject for Chapter 12.)
At 14:07 Riley changes catalog’s image line back to v63, and the pipeline writes it into the record. What does Kubernetes do?
Answer the question above to see what Kubernetes did
14:01
Record
14:04
Record
~14:08:30
Record
The same ReplicaSet, kept and reused
Rolling back is a change to the record like any other. Riley’s edit put v63’s description of each Pod back into the Deployment, and the Deployment loop saw what it always looks for: a gap between the version the record wants and the version its Pods run. It closed that gap in the usual way. It already had a ReplicaSet for v63, kept with no Pods since about 14:03, so it used that one, and, with the default settings, it filled it and emptied v64’s in the same gradual way it handles any change of version. That’s why going back took about a minute too, and why it needed the same spare room.
Declaring an earlier version again like this is a rollback.
New term · 04 / 04
rollback
Declaring an earlier version again. The Deployment rolls it out like any other change, from the ReplicaSet it kept, if it still has one.
It changes what runs, not what already happened.
Notice what the loop didn’t need to know: that this was an “undo”. It keeps no list of the steps it took at 14:02 to replay backwards. It compares the latest record with what is running now, and heads for the record; engineers call this level-based behaviour, because the loop acts on the gap it sees, not on a history of what happened. If the record had changed again halfway through, the loop would simply have headed for the newer version instead.
The ReplicaSets a Deployment keeps are also its history. By default it keeps the last ten old ones, emptied, and the command engineers use to undo a rollout (below) can step back only through those: once one is deleted, that version is gone from the history. Declaring an older version in the files, as Riley did, reaches further back than that, because the loop simply makes a new ReplicaSet for it, as it would for any new version, as long as the image is still in the registry (§08).
Undo with a command, or change the file back?
Engineers can roll back in two ways. One is a command, kubectl rollout undo, which changes the cluster’s record directly. The other is to change the image line back in the team’s files and let the pipeline apply them, as Riley did. The command is quicker, but it leaves the files still naming v64, so the next time the pipeline applies them, v64 rolls out again. Kubernetes’ documentation warns against managing one object both ways. Changing the file back is the declarative way to roll back, and GitOps tools (Chapter 13) make it the only way.
§ 08 What going back assumes
Riley’s rollback rested on two assumptions, and on Thursday both held. The first is that the old image still exists. The kept v63 ReplicaSet holds the image’s name, catalog:v63, not the image itself: the image lives in Fennel’s registry, and a node downloads it from there when it doesn’t already have a copy. If the image had been deleted, a rolled-back Pod could have started only on a node that still had a copy; anywhere else it would have waited, retrying, and the rollback could have stalled like any bad rollout. Fennel’s registry deletes catalog images after 30 days to save space, so Riley’s team can go back about a month, and no further.
image: catalog:v63a tag: a name that can be moved
image: catalog@sha256:9d2e…41a7a digest: names one exact image, forever
- catalog:v64sha256:5b07…c2e9
- catalog:v63sha256:9d2e…41a7← both lines name this image today
- catalog:v62sha256:e41c…07bd
- catalog:v61sha256:2a9f…d310
- older catalog imagesdeleted after 30 days
Fennel’s pipeline never reuses a tag
The second assumption is that v63 still means the image it meant at 10:15. An image never changes once it is built (Chapter 2), but the tag that points at it can: a tag such as v63 or latest can be moved to a different image, and Pods started from it at different times would then run different code. A digest, the image’s fingerprint, always names one exact image. Fennel’s pipeline never reuses a tag, and that is what makes “go back to v63” mean the same v63.
Beyond those two assumptions there is a limit that no amount of care removes: rolling back changes which version the Pods run, and nothing that the bad version already did. Twenty-three orders went through at $0.00 between 14:02 and 14:08, and they stayed that way, as would any emails sent or records written in those minutes. Jordan and Morgan decided to honour the orders. Data that has to outlive the Pods that wrote it is the subject of Chapter 10.
Each of these points turns into a question for Jordan’s list:
- “Do we keep old images for as long as we might need to go back, and do our tags ever move?”
- “If a bad version runs for six minutes, what will it have written that rolling back won’t fix?”
§ 09 The loops so far
The Deployment loop joins the loops from Chapters 4 and 5. It doesn’t replace any of them: it works through the ReplicaSet loop, which works through the scheduler and the node agents. Fig. 07 shows all of them through the loop lens, with what this chapter added.
| Loop | Wants | Sees | Does | Sam’s job, now a loop’s |
|---|---|---|---|---|
| The memory kill itself is the operating system’s, on the node; the node agent restarts the container afterwards. | ||||
§ 10 The conversation
On Friday morning, the team meets to review Thursday’s incident. Riley, a little defensive, speaks first.
“The rollout passed health checks, so it wasn’t the deploy.”
Riley
At Friday’s review of Thursday’s incident, Riley says: “The rollout passed health checks, so it wasn’t the deploy.” What would you ask Riley, and why? Think of your questions, then show the answer.
ANSWER“What do our health checks actually test? Would they catch wrong behaviour, or only crashes?”
catalog’s readiness check passes when the app is up and can reach its database, and a check passes on a success code alone, so a version that shows $0 passes by design. “Passed health checks” only proves that v64 started and could reach its database. It was the deploy: the rollout did exactly what it is built to do, and it is built to protect availability, not correctness. A rollout that completes has proved only that every new Pod passed its checks, not that the release is good.
Then, to follow up:
- “What would have caught wrong prices before every customer saw them: a check that tests a real price, or a release that goes to a small share of customers first while something watches, a canary?”
- “Who noticed, and how fast?” Morgan noticed from the orders after about three minutes. What would have noticed without Morgan?
- “If v63 had been bad too, could we have gone back further? Are old images kept, and do our tags ever move?”
- “And what do we do about the 23 orders?” Rolling back didn’t change them.
Noticing wrong behaviour sooner is where Chapter 12 picks up.
In short
- A Deployment is a record of which version of a service should run, and how many copies. Its loop keeps one ReplicaSet per version and shifts copies between them a few at a time, keeping enough Pods ready throughout (with three copies and the default settings, an old Pod goes only once a new one is ready), which needs spare room while both versions run.
- The readiness probe decides whether a Pod is ready for traffic; the liveness probe decides whether it is still alive, or has to be restarted.
- A version whose Pods crash on start or never become ready stalls the rollout, with the old version still serving. A version that passes its checks and does the wrong thing rolls out completely.
- Rolling back is declaring the old version again. The same loop rolls it out the same way, from the ReplicaSet the Deployment kept if it still has one, as long as the old image is still there.
- Rolling back changes what runs, not what the bad version already did.
All through this chapter, “only ready Pods are sent traffic” was the work of a loop we haven’t named. Every release gives catalog three new Pods, with new names and new addresses, and storefront still has to find them, at once, and only the ready ones. Chapter 7 is about how callers find a service whose Pods never stay put.