Finding each other: services and traffic

§ 01 Year 3: who passes on the addresses now?

Think back further, to 03:04 on a Saturday in Chapter 3, before Fennel had Kubernetes. Machine 03 died and took orders with it. Sam started orders again on machine 04, the only running machine with room, but customers still couldn’t place an order, because storefront’s settings still named machine 03, and storefront went on sending orders there until Sam passed on the new address. Connecting the services to one another was one of Sam’s jobs, and Sam did it by hand, at three in the morning.

On Kubernetes, addresses change far more often than machines die. Every replacement gives a service a new Pod at a new address, and every release gives it new Pods too: Riley’s team ships catalog two or three times a day, with three new Pods each time. Yet nobody passes on an address any more. Since storefront moved into the cluster, it has reached orders and catalog through every one of those changes without anyone touching its settings. That includes 03:10 on a second Saturday, in Chapter 4, when node 03 died again, this time on Kubernetes, and orders-q2rw replaced orders-5c8x.

Jordan“Sam used to pass on the new addresses. Who does that now?”

Ari“Nobody. storefront doesn’t call an address any more. It calls a name.”

There has been one wrinkle. On the Tuesday in Chapter 5 when orders ran out of memory three times and was restarted each time, customers saw errors for a moment at about 12:40, 18:50 and 20:10. Ari’s incident note adds what storefront’s log showed at each of those moments: connection refused, from orders. Jordan read the note twice.

“If the name always finds orders, what refused us?”

Jordan

To answer that, start with how storefront finds orders at all.

§ 02 A list copied by hand

Every Pod is given a network address of its own when it starts. Before Kubernetes, Sam passed on addresses like these by hand. Try the same on the cluster, with catalog on the Thursday of Chapter 6’s releases.

§ 03 One name that stays put: the Service

When Ari moved each of Fennel’s services into the cluster, they wrote one more short record beside it. For orders, the record gives a name, orders; a selector, app: orders, which says which Pods belong (any Pod carrying that label, and orders’ Deployment gives the label to every Pod it makes); and the port that callers connect to. Fig. 02 shows a shortened version of it, beside what Kubernetes adds.

Kubernetes then adds three things of its own. It gives the record an address, 10.96.0.25 (engineers call it the cluster IP), which stays the same for as long as the record exists, however often the Pods behind it change. The cluster’s own DNS answers the name orders with that address, which is why storefront’s settings say only orders. And beside the record, Kubernetes keeps a list of the addresses of the Pods that match the selector and are ready. Since 03:10 on that second Saturday (Chapter 4), the only Pod on that list has been orders-q2rw.

The record and what Kubernetes adds to it together make up a Service.

The word now has two meanings on this page, so it helps to keep them apart. At Fennel, orders is a service: a program the team writes and runs as Pods. Each of Fennel’s services also has a Service of the same name, the kind of record in Fig. 02. This chapter writes the program in code font, orders, and the record with a capital S, as in “the orders Service”.

The name a caller uses is a DNS name, the same kind of name as a website’s, except that only the cluster’s own DNS answers it. Inside Fennel’s cluster, orders on its own is enough.

§ 04 Thursday, 10:15, from storefront’s side

Go back to that Thursday’s 10:15 release in Chapter 6, when catalog went from v62 to v63 in about a minute. The Deployment loop started one v63 Pod, waited until it was ready, stopped one v62 Pod, and did that three times over, on nodes 05 and 06, where catalog’s Pods run. All the while, storefront kept calling catalog. W-01 replays that minute, and this time it adds what storefront was relying on.

At the top of the drawing, the panel for the cluster’s record now shows the catalog Service. At the top of its block are its name and its address, which is what storefront uses; under them is its list of ready Pods, each entry giving a Pod’s version and address. The other nodes are nearly full and nothing on them changes, so they’re left out, as in Chapter 6.

Keeping that list is the job of one more loop, which has been running in the control plane since the move, beside the loops from earlier chapters. It watches every Service’s selector and the Pods that match it, and it rewrites the Service’s list whenever one of those Pods becomes ready, stops being ready, or goes away. Engineers call the list the Service’s endpoints, and the loop the EndpointSlice controller, after the records that hold the lists; on this page it’s the endpoints loop.

W-01Fennel’s nodes, versions, addresses and timings are fiction. By default, a Service’s list holds only its ready Pods, and a Pod that is shutting down leaves it at once.

In Chapter 4’s terms, the endpoints loop is one more implementation of the same reconcile pattern, and its loop lens is easy to fill in. It wants each Service’s list to hold exactly the ready Pods that the Service’s selector matches. It sees every Service’s selector, the Pods that match it, and which of those Pods are ready. What it does about a gap is rewrite the list, and the change usually reaches callers within seconds.

This is the loop that put orders-q2rw on the orders list at 03:10 in Chapter 4, which is how storefront found the new Pod with nobody passing anything on. It is also the loop that, in Chapter 6, took a catalog Pod that had lost its database connection off the list while its readiness check failed, and put it back once the check passed. This loop only writes into the record: it changes the list, not where any call goes. How a call actually follows the list is the subject of §05.

It’s tempting to think that a Service keeps an eye on its Pods’ health, and stops sending calls to broken ones. It doesn’t. The endpoints loop copies each Pod’s ready or not-ready answer from the readiness check that the team wrote (Chapter 6), and does no checking of its own. So a Pod with no readiness check is listed as soon as its container is running, even if the app inside isn’t answering yet. And at 14:02 that Thursday, the v64 Pods that showed every price as $0.00 passed their check, so the loop listed them like any others.

The same list explains two more things from Chapter 6. Some customers saw $0 prices before the 14:02 rollout had finished, because for about forty seconds the list held v63 and v64 Pods together, and each new connection went to one of them, picked at random by default. It also explains why a canary release, which Chapter 6 described as sending a small share of customers to a new version first, takes more than starting one new Pod. With one v64 Pod among three on the list, roughly a third of new connections would reach it, on average, by chance. Sending exactly one customer in twenty to it takes routing rules that a plain Service doesn’t have.

§ 05 Where a call goes

Ari’s team likes anything that every order depends on to run in more than one copy, so that one failure can’t take it down. Every call from storefront to orders uses the orders Service. The Service has an address and a name, as a server does, and at Fennel the word “service” has always meant a program that runs. So it’s natural to picture the Service as a box in the middle, taking each call and passing it on.

§ 06 Tuesday, 12:40: connection refused

Now back to Jordan’s question. orders runs a single copy, orders-q2rw, on node 04. On the Tuesday in Chapter 5, at about 12:40, its memory reached its limit, the operating system killed its container, and the node agent restarted the container in place, as Chapter 5 showed.

§ 07 From a customer’s phone to a Pod

By default, a Service’s address works only inside the cluster. That’s all storefront needs to call orders. Customers’ phones, though, are out on the internet.

One way to let them in is one Service at a time. A team can set a Service’s type to LoadBalancer, and the cloud provider then creates a load balancer for that one Service: a machine that the provider runs, with a public address, which passes incoming connections on to the Service’s Pods. Each Service set up this way gets its own load balancer, its own public address and its own hourly charge.

(Engineers also say that a Service “load-balances”, as §05 mentioned, meaning the spreading done by each node’s rules. Here, a load balancer is the provider’s machine.)

The other way in serves many Services at once, for web traffic. The cluster gets one entry point: a set of routing rules that sends each request to a Service according to the name or path it asks for, such as fennel.example to storefront. The rules do nothing by themselves. A controller carries them out, either one that the provider runs or one that the team installs and keeps up to date, and in the usual set-up it sits behind a single cloud load balancer. The older form of this entry point is called an ingress; the newer one, which the Kubernetes project now recommends, is a gateway.

Fennel has used the second way since the move: one gateway, run with the provider’s controller, behind one cloud load balancer with Fennel’s one public address. So far the gateway has a single route, which sends fennel.example to storefront.

§ 08 The loops so far

The connecting job that Sam did by hand at 03:04 is now split between two loops: one in the control plane keeps each Service’s list, and one on every node turns the lists into rules. Fig. 06 shows both beside the loops from earlier chapters.

§ 09 The conversation

A few weeks later, the question of how to let callers in from outside comes back, this time from Riley’s team.