Who runs where: scheduling, resources and cost

§ 01 Year 3: the first bill since the move

Since 03:04, when node 03 died, Fennel’s cluster has run on three nodes: 01, 02 and 04. On Friday evening, as the dinner rush began, the dispatch team asked for twice as many copies of dispatch, and one of the new Pods didn’t start. Ari’s dashboard listed it as Pending for most of an hour, until Ari’s team rented two more machines and added them to the cluster as nodes 05 and 06.

On Tuesday, the orders Pod restarted three times between lunch and the end of the dinner rush. Each time, customers saw errors for a moment, and each time the dashboard gave the same reason: OOMKilled.

At the end of the month, Morgan opened the first full cloud bill since the move. Before the move, Fennel had given back the spare machines Sam used to keep and was running on four, and the new bill was higher than the last one from those days.

“We moved onto Kubernetes so our machines would be used better. Now we pay for five of them instead of four, and Ari’s dashboard says they’re less than a third busy, even on a Friday night. What are we paying for?”

Morgan

Jordan forwards the question to Ari, whose answer covers Friday and Tuesday as well.

“All three come back to two numbers our teams wrote down for every service when we moved.”

Ari

Ari’s answer starts with the loop that picked node 04 for orders after node 03 died, and it ends with the questions Jordan should ask before anyone changes those two numbers.

§ 02 What the scheduler counts

In Chapter 4’s drawings, every Pod had a size, every node had room for six units, and the scheduler looked for “a node with room”. Kubernetes didn’t measure those sizes, though. Each team wrote them into its service’s manifest when Fennel moved, declaring how much memory and how much processor each copy of the service should have set aside for it. The figure below shows the part of storefront’s manifest that does this. Engineers call these numbers the Pod’s requests. In it, storefront’s two units of room appear as its real memory request: 4 GB.

Each node, in turn, offers a fixed amount of both. Fennel’s nodes are rented machines with 16 GB of memory and 4 processor cores each. Part of every machine is kept back for its operating system and for Kubernetes’ own programs, so the Pods on a node can reserve only about 13 GB of memory and 3.9 cores between them.

Memory and processor are counted separately, and a Pod fits on a node only if both of its requests fit into what that node still has unreserved. The figure below shows why that matters: a node can have plenty of memory left while its processor is nearly all reserved, and then a new dispatch Pod doesn’t fit there, however much memory is free.

The loop that makes this check is the scheduler, the part of Kubernetes that gives each new Pod a node. It works in two steps. First, it rules out every node that doesn’t have room for the Pod’s requests. Then it scores the nodes that are left and picks the one with the best score, choosing at random if two are tied. With default settings, the score mainly rewards the node with the most room left unreserved. The scheduler also prefers, by default, to spread copies of the same service across different nodes, but only as a preference: if it can’t spread them, it places them anyway.

That’s why, in Chapter 4, when Ari applied storefront’s manifest to the new cluster, Kubernetes put its three Pods on three different nodes. Nodes 01 and 04 had the most room left, so the first Pod went to one of them and the second to the other. The third went to node 02, which had no storefront copy yet and more room than node 03. Notice what the scheduler wasn’t trying to do: fit Fennel onto as few nodes as possible.

Teams can also add placement rules of their own, such as keeping a service’s copies on different nodes, sending some Pods only to particular machines, or keeping ordinary work off special ones; your engineers will tune these.

The scheduler never looks at how busy a node is. When it checks for room, it counts what the Pods on each node have reserved, not what they are using. The Kubernetes documentation says the scheduler refuses a node whose reservations are full “although actual memory or CPU resource usage on nodes is very low”, and it gives the reason: the room is protected for when “resource usage later increases, for example, during a daily peak in request rate.” A reservation is a promise to the Pod, and the scheduler keeps it even while the Pod isn’t using the room. A reservation isn’t a cap, though: a Pod can use less than it reserved, or more, as long as its node has room to spare. What stops it, and what happens when the room runs out, come later in the chapter.

In Chapter 4’s terms, the scheduler is one implementation of the reconcile loop, and you can now fill in its loop lens completely. What it wants, the desired state, is every Pod on a node with room for its requests. What it sees, the real state, is Pods that have no node, and how much memory and processor each node has left unreserved. When the two differ, because a Pod has no node, it rules out the nodes that are too full, picks the emptiest of the rest, and writes that node into the Pod’s record. That leaves one question for the next section: what happens when no node passes the first step?

§ 03 Friday, 17:45: a Pod with nowhere to go

Since node 03 died, Fennel’s cluster has run on three nodes, 01, 02 and 04, and the Pods on them have reserved most of their room. Friday’s dinner rush is when dispatch, the service that assigns drivers to orders, is busiest. At 17:45 the dispatch team decides it wants six copies running tonight instead of three, and changes the number in dispatch’s manifest.

The following widget follows the next hour step by step as you scroll. It’s drawn similarly to how it was in Chapter 4, with one change: it now counts in real units. Each node’s column stands for the 13 GB of memory its Pods can reserve, each Pod’s height is its memory request, and memory that no Pod has reserved is drawn as dashed space. Processor is reserved separately, so it gets its own line at the foot of each node, counting the cores its Pods have reserved out of the 3.9 they can. During the scheduler’s steps, each node also shows its verdict for the Pod being placed: it fits, or the node is short of memory, of processor, or of both. The record and the loops work as they did in Chapter 4.

W-01Fennel’s requests and nodes are fiction. The scheduler’s reason is in the format engineers see; its exact wording varies between Kubernetes versions.

For most of that hour, the third dispatch copy was pending, and Pending is the word Ari’s dashboard showed for it.

Each time the scheduler fails to place a Pod, it records why. On Friday the line read 0/3 nodes are available: 2 Insufficient cpu, 3 Insufficient memory., which is in the format engineers see. Read it from left to right. “0/3 nodes are available” means that none of the three nodes passed the first step. The rest counts what each node was short of: all three lacked 2 GB of free memory, and two of them, 01 and 02, also lacked a whole free core. So the line already answers the first question to ask about a pending Pod: what is short, and where?

Notice what the line doesn’t say, which is that the cluster was full. Between them, the three nodes had 3 GB of memory free, more than the Pod’s request of 2 GB, but only in pieces too small for it; free room stranded like this is common on real clusters, and it matters to Morgan, because those leftover pieces on 01, 02 and 04 are room Fennel pays for that no Pod of this size can use.

There are two wrong ways to read Pending. It isn’t a failure: nobody has to resubmit anything, because the scheduler keeps trying and places the Pod as soon as a node has room for it. But it isn’t harmless either, because in Fennel’s cluster no loop makes room by itself. Someone has to add a node or free some room, as Ari’s team did at about 18:40. (Chapter 8 introduces a loop that can add nodes.) When a Pod is pending, these are the questions to ask:

• “What’s short, and on how many nodes?”
• “Is it short because demand really grew, or because what we reserved is more than we use?”

The second question leads straight to Morgan’s bill.

§ 04 What Morgan pays for

Fennel rents its machines from a provider, the cloud company that owns them. In the provider’s standard mode, which is the one Fennel uses, the bill counts machines: Fennel pays for each node for as long as it exists, whether the room on it is reserved, used or empty. So Morgan’s five nodes constitute the bill. Nothing in Fennel’s cluster rents or adds nodes by itself. Ari’s team decides how many machines to rent and adds them by hand (Chapter 8 introduces a loop that can). What the requests decide is how many nodes Fennel needs: on Friday, the Pods’ reservations didn’t fit on three nodes, so a Pod waited until Ari’s team added two.

The following widget works out how many nodes Fennel needs for a given set of memory requests. Like the scheduler, it places every Pod by its requests, with both of its numbers on one node; unlike the scheduler, it packs the Pods as tightly as they will go, so it counts the fewest nodes Fennel could rent. It starts with the memory requests Fennel’s teams have been using in this chapter, and keeps room for one more node so that Fennel can lose one. Each Pod’s outline is what it reserves; the fill inside is what it used at last Friday’s rush. There is no target. Some things to try:

• Set each request to what that service used at the rush. How many nodes does Fennel need now? Look at the processor line under each node: what stops the count going lower?
• Double payments’ request, as in the question above. Does the count change? How far can it go before it costs a node?
• Push orders past 13 GB. What happens?

W-02The widget packs Pods as tightly as they go, to count the fewest nodes; the scheduler, by default, spreads Pods out instead, so a real cluster needs at least this many. Each change places every Pod afresh, while a real cluster never moves running Pods. Processor requests, and the memory request of notifications, stay as the teams set them.

A Pod whose request is bigger than the room any node has for Pods is never placed: it stays pending for good, however long the scheduler keeps trying. The answer is a bigger node, a machine with more memory, which Ari’s team would rent and add by hand.

Fitting Pods’ reservations onto as few nodes as possible is called bin packing, and the widget above does it to count the fewest nodes Fennel could manage with. The scheduler, by default, doesn’t. It prefers the emptiest node and spreads copies out, which is safer for Pods but isn’t the cheapest arrangement. Packing tightly is a setting engineers can switch on, and some fully managed modes pack for you. Either way, the scheduler only places new Pods. It never moves running ones to empty a node, because Pods are replaced, not moved.

The widget’s “Room for Pods on the rented nodes” bar splits the room nobody uses into two kinds, room no Pod reserved (spare nodes, or pieces stranded too small for any Pod) and room reserved but not used, and the “less than a third busy” on Ari’s dashboard counts both.

Some setups bill per Pod instead of per machine. They charge for what each Pod requests, not for what it uses, so the lesson is the same: padding costs money.

Here is the answer to Morgan’s question: Fennel rents five nodes because the teams’ padded requests filled three nodes on paper while the services used far less, so Ari’s team rented two more. One of those, node 06, has sat nearly empty since, a node’s worth of spare room; whether to keep it is a choice, and the next section explains why Fennel might want to do that. Requests closer to what each service uses at its busiest would let Fennel rent fewer nodes, but not as few as possible.

§ 05 Room on purpose

Nodes go down, as node 03 did in Chapter 4. When one does, its Pods are replaced, and the scheduler has to place the replacements on the nodes that are left; if those nodes are full, the replacements wait for room, like the dispatch Pod at the dinner rush. Spare nodes are insurance against that: room kept free so that a lost node’s Pods have somewhere to go. The insurance is paid for a node at a time, and the “Keep room to lose a node” switch in the widget above shows what it costs; how much to keep is a decision for engineers and finance together.

“If we lose a node during the dinner rush, where do its Pods go?”

The question to ask

§ 06 Tuesday: when a Pod uses too much

On Tuesday, Ari’s dashboard showed the orders Pod, orders-q2rw, with a restart count of 3. Its last state read Terminated · Reason: OOMKilled · Exit code: 137. Engineers will recognise the 137, but what matters for you is the word before it.

A Pod can use more than it reserves, as §02 said, but not without end. A manifest can also set limits: the most memory and processor each copy may use on its node. Here is orders’ manifest with both: it reserves 6 GB of memory and may use up to 7 GB.

The chart below shows orders’ memory through Tuesday, against those two numbers.

Each time orders’ memory reached 7 GB, the operating system on the node stopped the container. Engineers say it was OOM-killed, from “out of memory”. The node agent then restarted the container, in the same Pod on the same node, so this was not a replacement like the ones after node 03 died, and the restart count went up by one. Each restart left a short gap in which customers couldn’t place orders. If the kills keep coming, the node agent waits longer and longer between restarts, just as it does with any crash-looping container, as we saw in Chapter 4.

Processor works differently. A container that reaches its processor limit isn’t killed. Instead it is slowed down to its limit (engineers say throttled), because processor time is handed out in slices that can be held back, while memory a program is already holding can’t be taken away. So the symptom of a processor cap is “it got slow”, and the symptom of a memory cap is “it restarted”.

§ 07 Using more than you reserved

Besides starting and restarting containers, as in Chapter 4, each node’s agent keeps an eye on how much memory its node has left. Ordinarily, Pods that use more than they reserved cause no trouble, because the node has room to spare.

If the node’s free memory falls too low, though, its agent removes Pods to protect the node and everything else running on it. That’s an eviction. The agent goes first for the Pods using the most memory beyond what they reserved, and Pods that stay within their requests go last. Unlike the kill in §06, an eviction removes the Pod itself. The ReplicaSet loop then writes a new Pod record, as it does for any Pod that disappears, and the scheduler has to place the replacement; on a full cluster, the replacement waits, like the dispatch Pod that waited at the dinner rush. So a request set below what a service really uses doesn’t save that service any room. It puts the service first in line.

The table below shows the loops you’ve met so far through the loop lens, with what this chapter added to each.

§ 08 The conversation

On Tuesday evening, after the third restart, Ari sends Jordan a message.

“We need to bump memory limits on orders.”

Ari