One app, one server

§ 01Friday, dinner rush

Fennel is a grocery-delivery startup with three people. Jordan and Morgan founded it, and Morgan handles the money. Sam is the only engineer, and built the one server the Fennel app runs on. In this chapter you’re in Jordan’s seat: you know roughly what a server and a deploy are, and you’ve never had to fix either.

It’s Friday evening, the dinner rush, the busiest hour of Fennel’s week. At 18:12 customers start seeing an error at checkout, and orders stop coming in. A few minutes later the first annoyed emails reach Jordan. Sam is at dinner across town and doesn’t pick up.

Here’s the strange part.

The disk had filled with log files, the running diary the app and its web server keep of every request. They had been growing for months, because nobody had set anything up to clear out the old ones. With no space left, the database couldn’t save new orders. Nothing already stored was damaged, but nothing new could be written.

Look at the last row, too. At 23:00 Sam set up a nightly job that deletes old logs, so the disk can’t fill up the same way again. It’s a good fix. Sam made it by hand, directly on the server, and didn’t write it down anywhere. That row comes back later.

§ 02What “the server” is

The server is one computer that Fennel rents in a data centre. Like a laptop, it has a processor, memory and a disk. Unlike a laptop, it runs day and night, and customers reach it over the internet.

The orange block is the Fennel app while it runs. A running program is called a process: the code, loaded into the machine’s memory, answering customers’ requests and writing orders and log lines to the disk. On Friday the process kept running. It just couldn’t do its job, because the disk it writes to was full.

When Sam ships a change, Sam copies the new code onto this machine and restarts the process. That copy-and-restart is a deploy, the arrow at the top of the figure.

So far the app sounds like code plus a computer. But code can’t run on just any computer. At the very least it needs the language runtime it was written for, such as Python, and that runtime is installed on the machine, not shipped with the code. What else does it need? You’re about to find out by hand.

§ 03What’s on Sam’s server, and can you rebuild it?

Below is Sam’s server as most people picture it: the code, running. First, work out what else is on it. Then try to build a second one using only Sam’s notes, and check whether it behaves like the first. It takes three or four minutes.

§ 04Why the notes fell behind

Everything you found on Sam’s server besides the code is something the app needs in order to work. Most of them are dependencies: the language runtime, the libraries and tools the code calls, and the config that tells it where its database is and which keys to use. The app also relies on the data on the disk, its orders and customers, which changes every minute. Many of these things live on the machine, not in the code, and a different machine may not have them. That’s how identical code broke photo uploads on your new machine.

Sam’s notes weren’t wrong when Sam wrote them. They described the server on the day it was set up. Then the photo tool went on in May, the email key in August, and the nightly log clean-up last Friday. Each was a sensible change, made by hand on the live server, and none of them reached the notes. Engineers call this drift (in full, configuration drift): changes to a system’s setup that go unrecorded. Every unrecorded change widens the gap between the notes and the server, so notes like Sam’s fall behind as a matter of course. Nobody has to be careless for it to happen.

Then there’s the holiday-sale setting. Sam changed it during a sale and can’t say whether it still matters. With no second machine to try it on, the surest way to find out is to remove it and see what breaks, on the server that takes every order. Martin Fowler, a well-known writer on software design, describes exactly this state of an old, hand-tended server: “You’re not sure what parts of the configuration are important.” That’s where “don’t touch it” comes from. Nobody knows which settings the app relies on, so every change is a gamble.

A server made unique by years of changes like these is a snowflake server. Fowler popularised the term in 2012: the result of all that hand-applied patching and tweaking is “a unique snowflake - good for a ski resort, bad for a data center.” Its first problem, he writes, is that it’s difficult to reproduce.

§ 05If this box vanished tonight

On Friday the server was sick, and Sam could nurse it back. Now suppose it’s gone.

So the length of an outage like this isn’t set by how fast someone can copy the code. It depends on how much is missing from the notes, and on whether the one person who knows the server is awake and reachable. Neither shows up in advance. The notes can’t list what they leave out, so the gaps surface as broken features. And the knowledge that fills them is mostly in Sam’s head, with some of it only on the machine that just died. That’s the practical meaning of Fowler’s warning about snowflakes: if the hardware fails, “it’s difficult to fire up another server to support the same functions.”

§ 06“Couldn’t we just…”

After Friday, everyone reaches for a fix.

Bigger machines also run out. There’s a limit to how large a single machine can get, and near the top of the range the price climbs steeply. Fennel is nowhere near that ceiling, so for now the real objection is Friday’s: a bigger server is still one server, built by hand. Engineers call a machine like this a single point of failure: when something goes wrong with it, the whole site can go down with it.

Sam’s idea uses a different kind of server. A virtual machine, or VM, is an isolated computer created in software, with its own processor, memory, disk and operating system. Several of them can run on one physical machine, and a new one can be created in minutes. The provider can also take a snapshot of the VM’s whole disk every night: an exact copy you can start again later.

VMs give you more machines, faster. They don’t tell you what should be on them.

§ 07The conversation

“We should keep a runbook for the server.”

Sam

A runbook is a written, step-by-step guide to setting up and repairing a system: think of Sam’s notes, kept current. It’s a good instinct. What matters is how Jordan answers.