8 ms·

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise record all that offline for later upload... because drivers probably can't even sign in. What's worse, though, is that this also implies that even if an outage is localized to a particular region, data for deliveries to that region isn't multihomed, so you can't just failover flex app service to another region and keep delivering. For a service that lives and dies on being able to deliver everything faster than everyone else... that seems like a massive oversight... but also completely in-character for Amazon.

GenerocUsername 5y ago
It happened once. My expectation of Amazon is that it would be fixed before it became a trend.
- midasuni 5y ago
  Nov 25 2020 https://www.datacenterdynamics.com/en/news/aws-us-east-1-region-suffers-errors-and-outages-impacting-its-status-page/ https://www.datacenterdynamics.com/en/news/aws-us-east-1-reg...
laluser 5y ago
Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and well, nothing ends up getting done.
- mdavidn 5y ago
  A one-day outage in December can be crippling for retail. I don't doubt that many functions are difficult to failover, but a bare-bones minimum seems straightforward. For example, evidence of delivery is append-only and only needs to be globally consistent later, after a dispute.
  - hef19898 5y ago
    I totally understand why Amazon halted everything. Sure, one could deliver shipments off-line and sort them out afterwards manually. And at a lower scale Amazon might have tried it (at least it would have been on the table when I worked there a couple of years ago). But what then? You had a complete loss of traceability of shipments and operations, once you regain it, a junk of shipments isn't there anymore where there are supposed to be. No you have not one potential root cause for this, the outage, that could be resolved by retriggering those shipments (not loss, as you didn't deliver anything to customers) but two: the outage and some off-line shipments. In case it was just one FC, sure that would be doable. If the whole network in a complete region goes down, no way to handle that. It is much easier and safer to just stop operations until the outage is resolved, re-route orders to other regions in the meantime, and then work through the backlog. Amazon's ops are good at that, specifically because they have almost complete transparency on their material flows. Going off-line would have jeopardized that transparency, making a quick recovery after the outage all the harder.
    - SpelingBeeChamp 5y ago
      Any idea if Amazon has insurance to cover this type of event?
      - hef19898 5y ago
        Can't speak for Amazon. Generally so I don't see how one could insure against it. I know that e.g. Allianz offers policies against IT outages. In that case so, what is the actual damage? Probably the delivery drivers paid without delivering and salaries, plus potential overtime to solve the backlog. Depending on the conditions a company the size of Amazon would get, maybe it's not worth it.

GenerocUsername 5y ago

It happened once. My expectation of Amazon is that it would be fixed before it became a trend.

midasuni 5y ago
Nov 25 2020 https://www.datacenterdynamics.com/en/news/aws-us-east-1-region-suffers-errors-and-outages-impacting-its-status-page/ https://www.datacenterdynamics.com/en/news/aws-us-east-1-reg...

laluser 5y ago

Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and well, nothing ends up getting done.

mdavidn 5y ago
A one-day outage in December can be crippling for retail. I don't doubt that many functions are difficult to failover, but a bare-bones minimum seems straightforward. For example, evidence of delivery is append-only and only needs to be globally consistent later, after a dispute.
- hef19898 5y ago
  I totally understand why Amazon halted everything. Sure, one could deliver shipments off-line and sort them out afterwards manually. And at a lower scale Amazon might have tried it (at least it would have been on the table when I worked there a couple of years ago). But what then? You had a complete loss of traceability of shipments and operations, once you regain it, a junk of shipments isn't there anymore where there are supposed to be. No you have not one potential root cause for this, the outage, that could be resolved by retriggering those shipments (not loss, as you didn't deliver anything to customers) but two: the outage and some off-line shipments. In case it was just one FC, sure that would be doable. If the whole network in a complete region goes down, no way to handle that. It is much easier and safer to just stop operations until the outage is resolved, re-route orders to other regions in the meantime, and then work through the backlog. Amazon's ops are good at that, specifically because they have almost complete transparency on their material flows. Going off-line would have jeopardized that transparency, making a quick recovery after the outage all the harder.
  - SpelingBeeChamp 5y ago
    Any idea if Amazon has insurance to cover this type of event?
    - hef19898 5y ago
      Can't speak for Amazon. Generally so I don't see how one could insure against it. I know that e.g. Allianz offers policies against IT outages. In that case so, what is the actual damage? Probably the delivery drivers paid without delivering and salaries, plus potential overtime to solve the backlog. Depending on the conditions a company the size of Amazon would get, maybe it's not worth it.