27 ms·
Amazon packages pile up after AWS outage spawns delivery havoc
- aschla 5y agoHere in Chicago every item on Amazon is showing an earliest delivery date of Saturday the 11th. Checked an item today I had bought yesterday and delivered overnight, the item is now showing available delivery as Saturday. What a terrible time of the year for this to happen to them.
- ryanchants 5y agoI'm in Chicago and just purchased a Prime-eligible book(but sold by a third party) that's listed as a Jan 6 delivery.
- blhack 5y agoExact same here in Phoenix.
- astrea 5y agoDoes Amazon host its core infrastructure in us-east-1?
- jdavis703 5y agoIt would appear so. Also it’s good Amazon has skin in the game here.
- riffic 5y agowhy wouldn't they?
- cbtacy 5y agoBecause it's terrible and has always been terrible? After 2017 I'll never use US-east-1 again. Hell... I should have learned that particular lesson in 2011 but it took two catastrophic failures for me to figure it out. There are numerous threads here on HN covering the topic "why does US-east-1 suck so hard." https://news.ycombinator.com/item?id=13756082 https://news.ycombinator.com/item?id=13756082 is just one example.
- bombcar 5y agoMaybe it’s terrible because they host their own stuff there.
- eyelidlessness 5y agoThat's the story of AWS as a whole.
- hinkley 5y agoI wonder if there’s too much moral hazard for them to run a data center just for Amazon but run exactly the same as the other data centers? Or would Amazon scope creep it into its own snowflake. I kind of doubt that they’re getting efficient use of hardware Amazon could be using but isn’t, since if someone allocates it it’s not available for Amazon anymore.
- jtdev 5y agous-east-1 is considered to be the least stable AWS US datacenter. Not sure if this has changed, but I believe it was because AWS uses us-east-1 as the “proving ground” for deployment of new services, systems, processes, etc.
- imwillofficial 5y agoIf their stuff was all super resilient and multi AZ, it would be easier to place blame on the customer
- haggy102 5y agoMulti AZ != Multi region. AWS has many services that run in multiple regions but some of their services are not.
- imwillofficial 5y agoCorrect, I mean region. Thank you
- sp332 5y agoYes, and also: [9:37 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. https://status.aws.amazon.com/ https://status.aws.amazon.com/
- shepherdjerred 5y agoSome of it. Lots of stuff is distributed, but us-east-1 hosts some global internal services
- NaturalPhallacy 5y agoEast was the first region. Salty programmers avoid it when possible. Because it's the oldest and cruftiest region.
- justinzollars 5y agoChaos.
- deleted 5y ago[deleted]
- throwaway888abc 5y agoThose Increasing error rates with some customers affected... /s All green https://news.ycombinator.com/item?id=29473630 https://news.ycombinator.com/item?id=29473630
- EamonnMR 5y agoYeah "increased error rates" is an understated way to describe a massive outage that took several services offline for a full business day.
- InsomniacL 5y ago'Patient is experiencing a decrease in heart-rate'
- LAC-Tech 5y agoI thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?
- andrewguenther 5y agoRegion outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.
- sneak 5y agoI thought the guidance was to be multi-AZ, as an AZ is the failure boundary?
- sitharus 5y agoAZs are physical boundaries, but the networking and software is interconnected. Regions are (mostly) isolated, though global services like IAM and CloudFront often have their main control plane in us-east-1
- ABeeSea 5y agoMulti-AZ deployment and multi-region failover is considered best practice.
- dharmab 5y agoAZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure. At a previous job where we needed to always be up, our disaster recovery plan assumed that the us-east-1 site had been hit by a meteor (not literally, but that's how we explained it to each other to put ourselves in the mindset.)
- deleted 5y ago[deleted]
- riffic 5y agocorrelation and causation, anyone? You'd think if it's anything mission critical, Amazon would be following its very own Well-Architected Framework. The Reliability pillar would speak to this.
- Hamuko 5y agoI don't think the Well-Architected framework specifies that you should use multi-region availability, but it's certainly a mentioned option for mission critical applications. Usually the go-to doctorine for high availability in AWS documentation is multi-AZ, not multi-region. https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/multi-region-scenarios.html https://docs.aws.amazon.com/wellarchitected/latest/reliabili...
- jjoonathan 5y agoI placed a few orders this morning. They appeared to go through, but didn't show up in my order list until this evening. Good thing I made the connection to the outage before I decided to try re-ordering! Sounds like others weren't so lucky.
- walrus01 5y agoI've seen mention of a major cable tv operator/last-mile residential ISP that had their entire field-tech dispatch system go down (based on a web gui https app that people can use on their phones) because it's hosted in AWS.
- Consultant32452 5y agoMy friend lives in very cold Alaska. She has an app on her car that lets her remotely warm her car, which is a requirement before driving because it's so cold. The AWS outage meant her phone couldn't talk to her car. She was 20 mins late to work because of that. This is the future. AWS has an outage and your car won't work.
- toomuchtodo 5y agoCouldn’t remote start the car with the fob? Viper/Directed has a paid cloud/app service for convenience, but the primary/default method to activate the remote start is RF.
- bserge 5y agoParaphrasing that stupid Apple ad: What's a key fob?
- hattmall 5y agoCouldn't you also just go outside and turn on the car, like people did before remote starts.
- throwawayboise 5y agoWe apparently don't teach advanced problem-solving like this in school anymore.
- faeyanpiraat 5y agoThats a funny comment, but in reality the heating was probably timed, and she only realized the car was cold when it was time to leave.
- ABeeSea 5y agoIf this outage isn’t the catalyst to get the Amazon side of the house to finally move out of US-east, I don’t know what will be. Or at least be multi-region. Although the cost to make all of Amazon commerce, logistics, and digital truly multi-region is probably an order of magnitude more than the impact of this outage.
- cbtacy 5y agoTrue. But let's be honest... this is neither the first such outage in US-east-1 nor the last. So I'd argue it's long past time for Amazon to pay the bill and go multi-region.
- ABeeSea 5y agoLast big e-commerce outage I can think of was Prime day years ago when Sable browned out. But that was non-AWS infrastructure. There was also the infamous s3 fat-finger; but I can’t think of a holiday shopping and delivery day with a massive outage.
- hinkley 5y agoI wonder if their system would allow them to move availability zones between regions. They could create new stealth AZs, move their stuff into them, then use those to start building a new region only they use. But I suspect there are third part integrations that benefit from being on the same AZ as Amazon APIs. I’ve been having little convos all day about how I think the inter region pricing creates a perverse incentive that’s exacerbating the us-east-1 situation.
- slimsag 5y agoCheck out the Amazon Flex Drivers subreddit[0], there are tons of people saying they were paid for the day of deliveries without delivering any packages and told to go home. [0] https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/im_not_messing_with_amazon_flex_today/ https://www.reddit.com/r/AmazonFlexDrivers/comments/rb3ggn/i...
- authed 5y agoI read many comments and didn't see one that was saying that but you might be right as I was too lazy to read everything.
- EarlKing 5y agoWouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise record all that offline for later upload... because drivers probably can't even sign in. What's worse, though, is that this also implies that even if an outage is localized to a particular region, data for deliveries to that region isn't multihomed, so you can't just failover flex app service to another region and keep delivering. For a service that lives and dies on being able to deliver everything faster than everyone else... that seems like a massive oversight... but also completely in-character for Amazon.
- caffed 5y agoI thought this story was about this house x-) "Neighbors in Tennessee city worry as Amazon packages pile up outside home" https://www.youtube.com/watch?v=qVQjEB2sxBw https://www.youtube.com/watch?v=qVQjEB2sxBw
- adventured 5y agoWhen a residential house becomes a review farm.
- jaxx 5y agowhen the house is becoming an amazon warehouse. This is crazy, i can't believe what the home owners thought of so many many boxes coming there.
- hallway_monitor 5y agoThere was a similar story on my local news as well. Someone's front yard filled with hundreds of packages.
- oldsecondhand 5y agoMore like dropshipping gone wrong.
- ethbr0 5y agoI don't even understand what's going on with this one. Transcribing the blurb from the clip, from interviewing the residents of the house: "[The people who live in the house] have a friend who has a contract with an Amazon warehouse in China. They say whenever that friend's contract expires, she will send the packages to their house for the family to sort and then send back to Amazon for the company to sell."
- deleted 5y ago[deleted]
- adanto6840 5y ago
- nimos 5y agoYeah my packages didn't get delivered today. I assume there will be cascading delays for the rest of the week. Getting close to xmas travel times, trying to finish up my shopping this week.
- deleted 5y ago[deleted]
- moneywoes 5y agoWill Amazon compensate all of their clients?
- andrewia 5y agoIf there's an SLA or reputation to uphold, probably, and otherwise they probably won't. So big AWS customers with SLAs will probably get compensation. And Amazon dot com customers that complain might get some compensation.
- joelbondurant 5y agoAWS is the McDonald's of computing.
- deleted 5y ago[deleted]
- faridelnasire 5y agoAs bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...
- nodesocket 5y agoSource? I'd be surprised if Google does not dog food. Though, I guess when GCP had their recent global load balancer outage but neither Gmail or Google search went down maybe not.
- shadowgovt 5y agoMost of Google's stuff runs on Borg, which predates GCP (and is the fabric GCP runs on top of).
- nodesocket 5y agoBorg is the predecessor to Kubernetes (sort of) right? They could have switched to Kubernetes and run on Google Kubernetes Engine (GKE).
- shadowgovt 5y agoI'm a bit out of the loop, but the vibe for years was "It isn't broke, so we're not going to rewrite it to run on something else." Actually, it's a bit more than that. Some half-decade ago, Google Cloud got stung by a coordinated attack over the holidays where attackers used stolen credit cards to build a net of GCE instances and do an attack on Tor via endpoint control. Cited by SRE in the postmortem was the relative immaturity of the cloud monitoring, logging, and "break glass" tools that SREs were accustom to in Borg... Essentially, Cloud didn't have the maturity of framework that Borg did and they felt the extra layer of abstraction complicated understanding and stopping the abuse of the service. This report had a chilling effect internally. Whereas management had previously been encouraging people to migrate to Cloud as quickly as possible, after this incident software engineering teams and the SREs that supported them were able to push back with "Can we trust it to be as maintainable as what we already have?" and put cloud on the defensive to prove that hypothesis.
- rasz 5y ago"If it bleeds we can kill it" Imagine ransomware targeting Amazon and spilling into the real world logistics, how much ransom could they charge.
- karmicthreat 5y agoWhenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.
- notfromhere 5y agoUs-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies
- jaxrtech 5y ago> Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies Was it Metra or the CTA out of curiosity?
- phantomread 5y agoIn Chicago I believe “commuter rail” almost always refers to Metra.
- kennydude 5y agoWhy is a commuter rail travel system dependent on AWS? O___O
- smsm42 5y agoSomething I'm not getting there. OK, one zone went down in AWS. But there supposed to be multiple ones? And if somebody knows how to use AWS so that one zone going down wouldn't knock over the whole thing, it should be Amazon? I thought the whole point of this multi-zone cloud setup would be so that even if one zone goes down, it could survive and perform as usual - maybe a little slower for a bit while backups kick in and such, but after initial short period everything should be fine? Here we're talking about the whole system going down hard. Was the problem much wider than reported - i.e. affecting all (US) zones at least, or is it that Amazon doesn't know or doesn't care to use its own infrastructure to build a robust system?
- DantesKite 5y agoIt’s beginning to sound like a cyberattack to me.
- tekno45 5y agoThis zone hosts the infra to manage the rest of the zones. DNS management is also here and can easily cause a global failure.
- matteotom 5y agoThis affected on region, us-east-1, which is the oldest and largest aws region, and also hosts many core aws services. Each region has several availability zones (AZs) that are basically whole datacenters. In theory AZs should be mostly isolated, but as we saw bugs happen. Reading between the lines of the status updates, it sounds like this either affected some core infra shared between AZs, or was from a change rolled out safely previously (maybe last night) and failed later (eg due to increased load in the morning). At the end of the day, building redundant systems is expensive (and may introduce whole new bugs), so they probably did the math and figured the risk of a whole region outage was less than the cost of building redundancy in some systems that are hard to make redundant.
- smsm42 5y agoThe whole concept that there are "core AWS services" living in one single zone sounds like anti-thesis to everything AWS should be about. What's the use of having all this nice distributed setup - and paying for it! - if a single failure in a single zone takes everything down anyway? I mean sure, maybe they built it on a shoe-string budget years ago - but since then, they had years and billions, and still didn't bother to fix it?
- 14 5y agoWeird I just ordered a package about 3-4 weeks ago and it didn’t show up so finally went to track it on amazon last week and got a message saying that they had lost the package and to request a refund. It kind of sucks because that was my mom’s Christmas gift now I have to figure out a plan B.
- kingcharles 5y agoI have this happen constantly through Amazon. Now all the hassle is on you to get the refund. And they won't process the refund immediately, so you have to pay out of your own money to get a replacement. And then often, as I've found, the price of the item has now gone up too, so you have to pay extra. p.s. Your username?! I can think this must be the only site you've managed to get that handle?
- 14 5y agoIt is the only site. Someone once had an even lower number and someone made mention to it and the person replied saying it was easy and he had just gotten it so I decided to try and see what was available. I had been a long term lurker and never posted but once I got this username I try and respectfully post when I can.
- kingcharles 5y agoWhat's fascinating is that it was still available in 2018. And the username I use here isn't available on pretty much any other site, but was still available here in 2021.
- noisy_boy 5y agoI think being "multi-cloud" is going to be a feature more prominently displayed in days ahead.
- phamilton 5y agoThe kinesis outage in 2020 and the S3 outage in 2017 didn't change much. Both as bad or worse than today.
- midasuni 5y agoUs easy 1 has an uptime of 99.9%, that’s low enough to get most sysadmins fired, but being able to point to headline news of how the same outage affects Amazon placates management. The key benefit of the cloud is blameshifting. It’s someone else’s problem, you just get the day off.
- mrich 5y agoThis really tarnishes any trust you might have had in Amazon as a professionally run business. (of course outages can happen and are to be expected especially at Amazon's scale, it's just the bad communication and the amateurish non-redundant setup of their own core services that is shocking)
- errcorrectcode 5y agoFYI: My local Amazon Hub+ caught-up in under an hour.
- reacharavindh 5y agoIt’s kind of ironic to go through AWS well architected framework and add all the complexity associated with multi-region setups when Amazon themselves couldn’t get it right. To be clear, I am not advocating for ignoring high availability setups. Just highlighting the complexity cost of it.
- omgitsabird 5y agoDoes Amazon follow their own multi-zone best practices? To be fair, we only run services in us-east-1, and we had zero downtime from this issue. The only issue we encountered were API calls that were failing for us, and those were for CloudWatch and CodeDeploy. We are heavily reliant on EC2, OpenSearch, and S3.
- floor_ 5y agoI feel like everything being centralized completely misses the point of the internet.
- coldcode 5y agoShit happens. My last employer is now mostly on AWS, and mostly in that region, so things broke all over the place yesterday (oddly enough the only service that is multi region was auth). In the past when we were mostly in our own datacenters we suffered as well, so that's not better either. One time a bad DNS update killed every single service worldwide and it took most of a day to properly restart thousands of services. Build complex systems with thousands or millions of things and it's bound to blow up sometime.