The Loadville Courier

At 23:48 on Sunday, Martha Keller, the night duty officer at the town data centre, was finishing her third coffee when every screen in front of her turned red at once. The pub on Mill Lane could no longer see which tables were taken. The cinema couldn’t sell a single ticket for the late show. The taxis stopped taking calls.

First she checked the main thing — the town’s database across the river?In real lifeDynamoDB, us-east-1Amazon’s cloud database in the Northern Virginia region., where nearly every shop and office in Loadville keeps its data. The database was working: every record in place, not a single error.

It was just as if nobody could see it.

The town’s computers aren’t supposed to know where the database lives. It has many entrances?In real lifeDynamoDB load balancersServers that take requests to the database and pass them on., some opening, some closing, so every few minutes each shop’s computer asks the town directory?In real lifeDNS (Route 53)The system that tells you at which address to find a service by its name.: “Where is the database now?” — and goes where it’s told.

The directory isn’t kept by people but by programs. One draws up fresh lists of entrances?In real lifeDNS planA set of entrance addresses with weights: where to send how much.. Three more — updaters?In real lifeDNS Enactor × 3Programs that write finished plans into DNS. — write those lists into the directory, one updater in each of three machine halls?In real lifeAvailability ZonesSeparate data centres within one region.. Keeping just one won’t do: if its hall goes dark, nobody updates the directory, and the entrances change all the time — the town’s computers would be walking up to closed doors. So there are three updaters, each in its own hall: if one goes dark, the other two carry on.

That night one updater got stuck. While it sat stuck, dozens of new lists came out. Just before, it had managed to check: “My list is newer than the one in the directory.” The check was honest, but by the time it came to, it was long out of date.

Then two things happened at once — in the very same second. The stuck updater came to and wrote its hopelessly old list over the newest one. And another updater had only just written that very newest list — and, as after every write, started the tidy-up?In real lifeclean-up of old plansDeleting old plans so they don’t pile up.: throw out old lists that have fallen far behind, so they don’t pile up.

The tidy-up didn’t look at what was in the directory right now — it threw out everything far behind the list just written. That included the list sitting in the directory: it was dozens of editions behind. There was no other list there — and the line “Database”?In real lifedynamodb.us-east-1.amazonaws.comThe name every program uses to find the database., which holds the answer to “where is the database?”, was left empty.

The result was strange: the database was working, but to the question “where is it?” the directory answered with silence.

At first Martha didn’t believe it. She re-ran the check, called the shift lead, then his boss. The automation that usually mends such things on its own got hopelessly tangled this time and stopped. The line had to be written back by hand, in the middle of the night: two engineers on the phone and a third in pyjamas. At 2:25 the line was back. For another quarter of an hour the town’s computers lived on the old empty answer in their memory?In real lifeDNS cacheA computer remembers the directory’s answer for a few minutes and doesn’t ask again., until they asked the directory again. At 2:40 the pub could see its tables again.

Morning

By morning it turned out the night wasn’t over. Nearly all of Loadville’s servers?In real lifedropletsPhysical servers on which EC2 machines run. are rented out by a single hire office?In real lifeDWFMAmazon’s system that manages the physical servers behind EC2. — each on a lease?In real lifeleaseA regular confirmation that a server is looked after and new machines can be placed on it. that is renewed through that same database. Overnight the leases ran out — and there was nowhere left to start new servers. As soon as the database was found, the office rushed to renew every lease at once, couldn’t keep up, started over — and stalled. By 5:30 it was back on its feet.

The servers began to come up, but not all the way. To be reachable, a server needs network settings — something like an address and a road. And the service that hands them out?In real lifeNetwork ManagerHands out network settings to new servers. had built up a huge queue overnight and was answering hours late. The server works, but there’s no road to it yet.

Then the watchdog programs?In real lifeNLB health checksLoad balancer checks on whether the servers behind it are alive. stepped in. Their job is to check regularly whether a server is alive and take dead ones out of service. They knocked on servers that had no road yet, heard an answer, then silence — and began throwing healthy servers out, then bringing them back a minute later. The town flickered.

At 9:36 the watchdogs were told not to throw anyone out on their own. At 10:36 new servers were starting properly again. By two in the afternoon the town breathed out.

What wouldn’t have saved it

• More updaters. There were already three, in three halls. Spares save you when a hall goes dark, not when two do the same job at the same moment.
• Checking more carefully. There was a check. The trouble is the time between “checked” and “written”: the world changed in between.
• A second database in the same town. Its address would sit in the same directory.

What did help was a copy in the next town?In real lifeglobal tables replicasA copy of the table in another Amazon region.: whoever kept a copy of their data there worked all night, even though the copy lagged behind.

What was done

• The programs that update the directory were switched off in every town — until the race is fixed and the directory learns to refuse wrong lists.
• The watchdogs got a limit on how many servers they may throw out at once.
• The hire office will practise getting back up after nights like this, and the service that hands out network settings will take new work only while the queue isn’t too long.

From the desk

• A number on every list. Write only if the directory still holds the list you saw when you started. If not, take a fresh one. A slow updater goes round once more, but spoils nothing.
• Don’t delete what’s in use. The tidy-up touches only lists that are no longer in the directory, and the line is never left empty.