Many containers, many machines: the orchestration problem

§ 01 Year 2: eight machines and a spreadsheet

It’s Year 2. Fennel’s app used to be one piece of software. It’s now four services, separate parts that each do one job: storefront (the site and app customers use), orders, payments and dispatch (which assigns drivers). Each service runs as one or more copies, and each copy is a container started from that service’s image. Fennel pays for about eight rented machines to run them on. Sam keeps a spreadsheet that says which container runs on which machine.

Jordan doesn’t see containers or machines. Jordan sees four things:

• Sam has been woken by a page at 3 a.m. more than once this month.
• Release days start with a message from Sam in the team chat: “Nobody deploy anything, I’m updating storefront.”
• The last time anyone needed the spreadsheet in a hurry, it was wrong.
• Morgan, reading the bill, asks why Fennel pays for eight machines when half of them look mostly idle.

Here is one week of Sam’s job, step by step.

§ 02 One week of Sam’s job

§ 03 What the jobs cost

Each of the six jobs in Sam’s week shows up somewhere Jordan can see it.

First, Morgan’s question: why do half of Fennel’s machines look mostly idle? When people place containers by hand, they play it safe. Moving a container later is slow and risky, so they leave free room on every machine and keep spare machines standing by. Some of that room is worth having: on Saturday night, the free room on machine 04 is where orders went. How much spare room is right is a question for Chapter 5. Still, idle machines are an old problem. One industry estimate, from data centres where each app had a server to itself, put the average server at only 12 to 18% busy.

Then there are the people doing the work. Google’s site reliability engineers have a name for manual, repetitive operations work that could be automated: toil. In their experience, toil grows in step with the service it supports, and too much of it burns people out, so they aim to keep it under half of each engineer’s time. Almost everything Sam did between Monday and Tuesday was toil.

And then there’s the spreadsheet. It’s Sam’s server notes from Chapter 1 again: a record kept by hand, of a system that changes every time something fails. A record like that falls behind, and on Saturday night this one did. It still says orders is on machine 03.

What if the record were the thing the machines act on, instead of a note about them? Hold that thought for the next chapter.

§ 04 Software that does the jobs

What would help Sam isn’t a faster Sam. It’s software that does the six jobs in Fig. 01 by itself.

Doing those jobs automatically, for as long as the containers run, is called orchestration, and software that does it is an orchestrator. An orchestrator doesn’t only act when someone releases a new version. At 3 a.m., when a machine dies, it starts the lost containers on other machines and passes on their new addresses.

You give an orchestrator a group of machines, physical or virtual, and tell it how big each container is. It fits the containers onto the machines itself. That group of machines, treated as one pool of room, is a cluster. The four machines Sam used in §02 would make a small one.

Both definitions say what an orchestrator does, not how it does it. How it manages all this with nobody at a keyboard is the next chapter’s subject.

§ 05 Its name is Kubernetes

The orchestrator this course is about is Kubernetes. It’s the name the course has been heading for since Chapter 1.

Kubernetes came out of Google. Google had run its own services in containers for years, on an internal system called Borg. It later built a second system, Omega, to try out new ideas, many of which went back into Borg. Just like Morgan at Fennel, Google cared about idle machines: one reason it built Borg was to let different kinds of work share the same machines, so that more of each machine gets used and costs come down.

You’ll hear that Kubernetes is Google’s Borg, released to the world. It isn’t. Kubernetes was Google’s third such system, built new with lessons from the first two, and open source from the start. Borg and Omega stayed inside Google.

Google published Kubernetes as open source in 2014. In 2015, at version 1.0, Google announced it would give Kubernetes to a newly formed foundation, the Cloud Native Computing Foundation, which is part of the Linux Foundation, so no single company owns it. By 2024, tens of thousands of people from thousands of companies had contributed to it.

Kubernetes isn’t the only orchestrator. Docker Swarm and Apache Mesos are two others, and Chapter 14 compares the options.

How Kubernetes does the six jobs with nobody at a keyboard is the next chapter. First, a name you’ll hear alongside it.

§ 06 Where Docker fits

You may already know the name Docker, and the last chapter on containers may have left you wondering how it fits in with Kubernetes. The two names come up together so often that they’re easy to mix up. So let’s introduce Docker properly, and see which job each one does.

A container is a general idea, not a product: the containers in Chapter 2 could have been built and run with any of several tools. “Docker” is the best-known set of those tools, from the company of the same name, and the one that made containers popular. On Riley’s laptop, docker build makes the storefront:v13 image, and docker run starts a container from that image, on that laptop. Docker’s tools act on the machine they’re run on.

Riley then pushes the image to the registry. From there, an orchestrator such as Kubernetes decides which machines in the cluster run copies of it, starts those copies, and keeps doing the six jobs for them. In 2015, Docker and other container companies set up the Open Container Initiative, and Docker gave it its image format. That format is now an open standard, so Kubernetes runs an image whichever tool built it. That’s why Docker and Kubernetes aren’t competitors in the usual sense: most teams use Docker’s tools to build images, and an orchestrator to run them across their machines.

Kubernetes was even announced at Docker’s own conference, DockerCon, in 2014.

§ 07 The conversation

After another 3 a.m. page, Sam brings a proposal to the planning meeting.

“We should move to Kubernetes.”

Sam

Moving still costs something: the platform, and the time to learn and run it. So the question for Jordan to bring to the planning meeting is: “Which of these jobs is hurting us most today, and is that worth the cost of the platform?” Chapter 14 comes back to the cost side.

That’s the end of Part I. Since its first server, Fennel has been catching up: the disk filled, so Sam wrote a clean-up job; a release broke in production, so the team moved to containers; a machine died, so Sam moved containers by hand at 3 a.m. Each fix came after the pain. Part II is about getting ahead of it: telling Kubernetes what Fennel wants running, and letting it keep things that way.