6 ms·

Kinda wish I still worked there and could see the post-outage writeup. My bet's on some unforeseen bottleneck that affects search and static pages. Almost ever

Kinda wish I still worked there and could see the post-outage writeup. My bet's on some unforeseen bottleneck that affects search and static pages. Almost everything within Amazon is crazy scaleable, but there are some bits where you scale them up and their behaviour changes radically. For instance, a service's cache misses might skyrocket as customers get distributed over a wider set of servers, causing service response times to increase just a little bit on average, tipping a dependent service over into more frequent timeouts, causing its downstream service to blow a timeout-percentage 'software fuse' and stop using that service... etc etc. Given that each of those services (and many more possibly-related ones) will have an on-call engineer paged into a conference call when the manure hit the rotating ventilation apparatus, there are going to be a lot of unhappy people cancelling their weekend plans right now. I definitely don't miss that aspect of the job!

adreamingsoul 8y ago
oncall is the worst
- cheeze 8y ago
  Not oncall for Amazon but I don't particularly mind it. It sucks to have to be near a computer for the time, but I actually find digging in and finding that needle in a haystack pretty fun. On a good team, the life interruption is minimal.
  - sateesh 8y ago
    As much as I like digging,debugging and learning that comes with it, fixing things under tremendous time pressure wears me out :(
    - iotscale 8y ago
      When I was on-call for a big project, I jumped every time my phone rang. It wasn't a good experience at all. "We are at the fair, I have 18 customers lined up but I can't take orders. Can you fix this now or should I start taking them on paper?" You know the joke that atheists don't believe in god until the plane nosedives? That was the moment for me :) "Oh, I... I need to see the error message before I can say anything. We don't have any errors logged. Hmm... Would you allow me to remote into your device?" "Can't do that. We lost the internet connection here, sorry." Ugh.
  - bpchaps 8y ago
    How long have you been on call for and how many 3am wakeup calls have you had? I used to think these same. Protip: you won't always be on a good team.
    - unityByFreedom 8y ago
      I'm in Asia and always thought I'd make a great on call person for those of you who haven't jumped ship. I should apply somewhere.
      - bpchaps 8y ago
        Hah. I'm genuinely surprised this isn't more of a thing.
- zerotolerance 8y ago
  I prefer to be oncall as long as I'm empowered to prioritize software improvements that reduce ops burden and improve customer experience.
- zwischenzug 8y ago
  We reduced the stress of on-call by introducing processes that supported creativity and reduced panic: https://zwischenzugs.com/2017/04/04/things-i-learned-managing-site-reliability-for-some-of-the-worlds-busiest-gambling-sites/ https://zwischenzugs.com/2017/04/04/things-i-learned-managin...
coryfklein 8y ago
Thanks for the comment and I love all the insight but serious question about: > there are going to be a lot of unhappy people cancelling their weekend plans right now. The week just started... are you saying that you can already anticipate the war room for an event like this to extend through this coming weekend?
- camtarn 8y ago
  Haha, no, oops. My last few days have been a bit crazy and I'm on holiday, so I just thought it was Sunday ;)
  - sigfubar 8y ago
    It's a good holiday if you can no longer remember what day it is.
    - devmunchies 8y ago
      But not good enough if you’re checking HN ;)
      - slap_shot 8y ago
        Do you seriously believe reading something like HN is work for its viewers??
        kajecounterhack 8y ago
        I think they meant that if you're on vacation you maybe shouldn't be looking at the computer.
        flattone 8y ago
        my memorial weekend on a few acre olympic rain forest camping trip: regular signal hunting for hn. Oddly found a 6x6 area which after a reboot with provide lte. after idle would have to reboot again to open some more links
throwaway427 8y ago
smile.amazon.com was working fine during the outage, if that helps narrow it down...
- deleted 8y ago
  [deleted]
- azhenley 8y ago
  It wasn't though. I saw an item being listed as a Prime Day deal on the normal Amazon site, then I searched for it through smile.amazon.com and it wasn't there. Went back to non-smile and it was there (but throwing errors so I couldn't click it anyway...)
  - Alex3917 8y ago
    I'm seeing this also. Same item is $169 on smile.amazon.com, versus $79 on www. Neither is working properly though.
- gnat 8y ago
  Not for me: a page that gave a fast 503 on amazon.com would spin indefinitely on smile.amazon.com
aviv 8y ago
My bet is they maxed out on the SQS FIFO 300 messages per second limit.
- camtarn 8y ago
  I have no idea really (I stopped working on the retail websites in 2013) but my gut feel is that that's about a couple of orders of magnitude too low at least. But, yes, unforeseen rate limits and size limits can cause many hilarious things to happen. I've seen a few good ones in my time. In particular, when somebody sets an upper size on an in-memory table and commits it thinking "Well, I added a few orders of magnitude safety margin - that should be enough for anybody", that's probably going to become an incident at some point in the distant future ;) With luck, the throttling or failure behaviour will only affect a few people, and it'll be spotted by looking at traffic graphs and noticing a very slightly elevated rate of service errors. If you're unlucky, though, when you hit the limit the whole service slows down, locks up, or just plain crashes, and something like this happens...
  - snewman 8y ago
    Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point. https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/ https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/
    - adreamingsoul 8y ago
      That's cool, but it's a bit more complicated then rate limits.
    - codebje 8y ago
      Don't forget to alert when the double derivative of capacity growth is above a (very low!) threshold, that can catch an explosive problem far earlier than the 70% mark, by which time it might be growing at 5% per second.
adrianmonk 8y ago
Weekend plans? Prime Day is going to be over in the early part of the week. Does Amazon expect a fix to be deployed ASAP after the immediate crisis is averted?
kaspm 8y ago
My guess was either that OR there wasn't backpressure on a cascading failover so as services failed they increasingly failed to more and more overloaded systems OR there WAS backpressure and it was the luck of the draw whether you were queued into an error page or got good data OR the autoscaling couldn't keep up with the onsale window. This used to happen in ticketing a lot. Ticketmaster has a talk somewhere where they talk about warming the scaling load and server cache in anticipation of big ticketing onsales. The time it took to autoscale was just too long.
- zerotolerance 8y ago
  The systems at Amazon are too large to leverage autoscaling. When I was there (in Marketplace) it was generally estimated that Amazon used about 80X the capacity of their entire AWS public cloud.
  - abrookewood 8y ago
    WTF? 80 times the capacity of their entire AWS infrastructure?? That seems insane.
    - caseysoftware 8y ago
      No, the exact quote was: > Amazon used about 80X the capacity of their entire AWS public cloud which is probably closer to "the capacity that is available via AWS is a tiny, tiny fraction of their overall computing power.. therefore adding it back in when things are falling over doesn't actually solve any problems."
      - Dylan16807 8y ago
        I don't understand the difference between the wording in your post and the post you're replying to.
        martin-adams 8y ago
HankB99 8y ago
When I checked it out, the site seemed responsive but search was broken. I searched "usb flash drive" and was presented with 2 results. I pondered this for a moment and then realized that there were 400 pages with 2 results/page (as far as I checked.) Perhaps there is some load shedding algorithm that reduces the result count per page to reduce load. It did discourage me from further searching so I guess it worked. ;)

adreamingsoul 8y ago

oncall is the worst

cheeze 8y ago
Not oncall for Amazon but I don't particularly mind it. It sucks to have to be near a computer for the time, but I actually find digging in and finding that needle in a haystack pretty fun. On a good team, the life interruption is minimal.
- sateesh 8y ago
  As much as I like digging,debugging and learning that comes with it, fixing things under tremendous time pressure wears me out :(
  - iotscale 8y ago
    When I was on-call for a big project, I jumped every time my phone rang. It wasn't a good experience at all. "We are at the fair, I have 18 customers lined up but I can't take orders. Can you fix this now or should I start taking them on paper?" You know the joke that atheists don't believe in god until the plane nosedives? That was the moment for me :) "Oh, I... I need to see the error message before I can say anything. We don't have any errors logged. Hmm... Would you allow me to remote into your device?" "Can't do that. We lost the internet connection here, sorry." Ugh.
- bpchaps 8y ago
  How long have you been on call for and how many 3am wakeup calls have you had? I used to think these same. Protip: you won't always be on a good team.
  - unityByFreedom 8y ago
    I'm in Asia and always thought I'd make a great on call person for those of you who haven't jumped ship. I should apply somewhere.
    - bpchaps 8y ago
      Hah. I'm genuinely surprised this isn't more of a thing.
zerotolerance 8y ago
I prefer to be oncall as long as I'm empowered to prioritize software improvements that reduce ops burden and improve customer experience.
zwischenzug 8y ago
We reduced the stress of on-call by introducing processes that supported creativity and reduced panic: https://zwischenzugs.com/2017/04/04/things-i-learned-managing-site-reliability-for-some-of-the-worlds-busiest-gambling-sites/ https://zwischenzugs.com/2017/04/04/things-i-learned-managin...

coryfklein 8y ago

Thanks for the comment and I love all the insight but serious question about: > there are going to be a lot of unhappy people cancelling their weekend plans right now. The week just started... are you saying that you can already anticipate the war room for an event like this to extend through this coming weekend?

camtarn 8y ago
Haha, no, oops. My last few days have been a bit crazy and I'm on holiday, so I just thought it was Sunday ;)
- sigfubar 8y ago
  It's a good holiday if you can no longer remember what day it is.
  - devmunchies 8y ago
    But not good enough if you’re checking HN ;)
    - slap_shot 8y ago
      Do you seriously believe reading something like HN is work for its viewers??
      - kajecounterhack 8y ago
        I think they meant that if you're on vacation you maybe shouldn't be looking at the computer.
        flattone 8y ago
        my memorial weekend on a few acre olympic rain forest camping trip: regular signal hunting for hn. Oddly found a 6x6 area which after a reboot with provide lte. after idle would have to reboot again to open some more links

throwaway427 8y ago

smile.amazon.com was working fine during the outage, if that helps narrow it down...

deleted 8y ago
[deleted]
azhenley 8y ago
It wasn't though. I saw an item being listed as a Prime Day deal on the normal Amazon site, then I searched for it through smile.amazon.com and it wasn't there. Went back to non-smile and it was there (but throwing errors so I couldn't click it anyway...)
- Alex3917 8y ago
  I'm seeing this also. Same item is $169 on smile.amazon.com, versus $79 on www. Neither is working properly though.
gnat 8y ago
Not for me: a page that gave a fast 503 on amazon.com would spin indefinitely on smile.amazon.com

aviv 8y ago

My bet is they maxed out on the SQS FIFO 300 messages per second limit.

camtarn 8y ago
I have no idea really (I stopped working on the retail websites in 2013) but my gut feel is that that's about a couple of orders of magnitude too low at least. But, yes, unforeseen rate limits and size limits can cause many hilarious things to happen. I've seen a few good ones in my time. In particular, when somebody sets an upper size on an in-memory table and commits it thinking "Well, I added a few orders of magnitude safety margin - that should be enough for anybody", that's probably going to become an incident at some point in the distant future ;) With luck, the throttling or failure behaviour will only affect a few people, and it'll be spotted by looking at traffic graphs and noticing a very slightly elevated rate of service errors. If you're unlucky, though, when you hit the limit the whole service slows down, locks up, or just plain crashes, and something like this happens...
- snewman 8y ago
  Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point. https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/ https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/
  - adreamingsoul 8y ago
    That's cool, but it's a bit more complicated then rate limits.
  - codebje 8y ago
    Don't forget to alert when the double derivative of capacity growth is above a (very low!) threshold, that can catch an explosive problem far earlier than the 70% mark, by which time it might be growing at 5% per second.

adrianmonk 8y ago

Weekend plans? Prime Day is going to be over in the early part of the week. Does Amazon expect a fix to be deployed ASAP after the immediate crisis is averted?

kaspm 8y ago

My guess was either that OR there wasn't backpressure on a cascading failover so as services failed they increasingly failed to more and more overloaded systems OR there WAS backpressure and it was the luck of the draw whether you were queued into an error page or got good data OR the autoscaling couldn't keep up with the onsale window. This used to happen in ticketing a lot. Ticketmaster has a talk somewhere where they talk about warming the scaling load and server cache in anticipation of big ticketing onsales. The time it took to autoscale was just too long.

zerotolerance 8y ago
The systems at Amazon are too large to leverage autoscaling. When I was there (in Marketplace) it was generally estimated that Amazon used about 80X the capacity of their entire AWS public cloud.
- abrookewood 8y ago
  WTF? 80 times the capacity of their entire AWS infrastructure?? That seems insane.
  - caseysoftware 8y ago
    No, the exact quote was: > Amazon used about 80X the capacity of their entire AWS public cloud which is probably closer to "the capacity that is available via AWS is a tiny, tiny fraction of their overall computing power.. therefore adding it back in when things are falling over doesn't actually solve any problems."
    - Dylan16807 8y ago
      I don't understand the difference between the wording in your post and the post you're replying to.
      - martin-adams 8y ago

HankB99 8y ago

When I checked it out, the site seemed responsive but search was broken. I searched "usb flash drive" and was presented with 2 results. I pondered this for a moment and then realized that there were 400 pages with 2 results/page (as far as I checked.) Perhaps there is some load shedding algorithm that reduces the result count per page to reduce load. It did discourage me from further searching so I guess it worked. ;)