6 ms·

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems

by snewman 10y ago

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it? Also, hindsight is 20/20, but before this incident it might have been hard to identify "full-scale restart of the index subsystem" as rising to the top of the list of things to test. One approach is to try to extrapolate from smaller-scale tests. It would be interesting to know what kinds of disaster testing Amazon does do, and at what scale, and whether a careful reading could have predicted this outcome.

pbkhrv 10y ago
> Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it? Rough guide: CT = cost of 1 full scale test with necessary infrastructure and labor costs added up CF = amount of money paid out in SLA claims + subjective estimate of business lost due to reputation damage etc PF = estimate of probability of this event happening in a given year if PF * CF > CT, then you run such a test at least once a year. Think of such an expense as an insurance premium. What Netflix does with their simian army is amortize the cost of doing the test across millions of tests per year and the extra design complications arising from having to deal with failures that often.
hollander 10y ago
Testing a full zone test is only possible when they have a new zone available, unused. I bet they do these test, and they now have a new scenario to test.
- snewman 10y ago
  They also probably have one or more test regions where they could perform a test like this. But it's presumably not at nearly the same scale as us-east-1, the region affected by this incident. And to a considerable extent the problem was one of scale. The writeup makes the recovery sound fairly straightforward; but due to the sheer size of S3 in this region, it took hours for the system to come back up, which was apparently unexpected. (Nit: this incident affected a region, not a zone. us-east-1 is a region, which is divided into zones us-east-1a, us-east-1b, etc. S3 operates on regions.)
twistedpair 10y ago
> Failure at every level has to be simulated pretty often to understand how to handle it Keep in mind, S3 "fails" all the time. We regularly make millions of S3 requests at my work. Usually we get 1:240K failure rate (mostly GETs), returning 500 errors. However, if you're really hammering an S3 node in the hash ring (e.g. Spark job), we see failures in the 1/10K range, including SocketExceptions, where the routed IP is dead. You need to always expect such services to die in your code, setting the proper timeouts, backoffs, retries, queues, and dead letter queues.
- Atheros 10y ago
  If you retry the get does it succeed usually?
  - twistedpair 10y ago
    Yes, retry has always worked (*us-west-2). Sometimes it's a 404 for an object written 1 sec prior, other times it's an S3 node that died mid request. Retry gets you to a different node.
grogers 10y ago
This is precisely why cells (alluded to in the write-up) are beneficial. If the size of a cell is bounded and you scale by adding more cells, testing the breaking point of the largest cell becomes an easier problem. There is still usually a layer that spans across all cell boundaries, which is what then becomes hard to test at prod scale (so you make that as simple as possible)

by snewman 10y ago

pbkhrv 10y ago

> Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it? Rough guide: CT = cost of 1 full scale test with necessary infrastructure and labor costs added up CF = amount of money paid out in SLA claims + subjective estimate of business lost due to reputation damage etc PF = estimate of probability of this event happening in a given year if PF * CF > CT, then you run such a test at least once a year. Think of such an expense as an insurance premium. What Netflix does with their simian army is amortize the cost of doing the test across millions of tests per year and the extra design complications arising from having to deal with failures that often.

hollander 10y ago

Testing a full zone test is only possible when they have a new zone available, unused. I bet they do these test, and they now have a new scenario to test.

snewman 10y ago
They also probably have one or more test regions where they could perform a test like this. But it's presumably not at nearly the same scale as us-east-1, the region affected by this incident. And to a considerable extent the problem was one of scale. The writeup makes the recovery sound fairly straightforward; but due to the sheer size of S3 in this region, it took hours for the system to come back up, which was apparently unexpected. (Nit: this incident affected a region, not a zone. us-east-1 is a region, which is divided into zones us-east-1a, us-east-1b, etc. S3 operates on regions.)

twistedpair 10y ago

> Failure at every level has to be simulated pretty often to understand how to handle it Keep in mind, S3 "fails" all the time. We regularly make millions of S3 requests at my work. Usually we get 1:240K failure rate (mostly GETs), returning 500 errors. However, if you're really hammering an S3 node in the hash ring (e.g. Spark job), we see failures in the 1/10K range, including SocketExceptions, where the routed IP is dead. You need to always expect such services to die in your code, setting the proper timeouts, backoffs, retries, queues, and dead letter queues.

Atheros 10y ago
If you retry the get does it succeed usually?
- twistedpair 10y ago
  Yes, retry has always worked (*us-west-2). Sometimes it's a 404 for an object written 1 sec prior, other times it's an S3 node that died mid request. Retry gets you to a different node.

grogers 10y ago

This is precisely why cells (alluded to in the write-up) are beneficial. If the size of a cell is bounded and you scale by adding more cells, testing the breaking point of the largest cell becomes an easier problem. There is still usually a layer that spans across all cell boundaries, which is what then becomes hard to test at prod scale (so you make that as simple as possible)