4 ms·

If architected correctly, stuff deployed into aws stays up during all but the most extreme outages.

by f0e4c2f7 4y ago

oars 4y ago
And then where do we discuss these extreme outages?
- shawnz 4y ago
  where did you discuss outages of HN's previous provider?
  - dralley 4y ago
    Less people are interested in outages of HN's previous provider.
- rubyist5eva 4y ago
  reddit.... shudders
  - mariojv 4y ago
    maybe not... https://aws.amazon.com/solutions/case-studies/reddit-aurora-case-study/ https://aws.amazon.com/solutions/case-studies/reddit-aurora-...
- staticassertion 4y ago
  If there's an extreme AWS outage, you're pretty much stuck to in-person or POTS.
- VoidWhisperer 4y ago
  HAM Radio
  - namecheapTA 4y ago
    People joke but from the top of a decent mountain I can reach 45 miles into the bay area from me on a $150 2m radio in my truck with rooftop antenna.
    - loxias 4y ago
      That is so cool. If I had that (truck+radio) I'd be very tempted to scatter a few cheap relay nodes at good locations in the bay area. Monkeybrains+comcast is quite reliable.
    - iasay 4y ago
      The problem with that is the only people you have to talk to is hams. (Ex ham)
      - nwh5jg56df 4y ago
        Well same for computers & the internet :)
        iasay 4y ago
        Fair point!
- mekster 4y ago
  Better to discuss the next day when it's over than seeing bunch of upset comments realtime posted.
  - atmosx 4y ago
    If the AWS status wasn’t a static HTML page, I would agree.
fulafel 4y ago
Of those "correctly" architected apps, most are not properly tested for the failovers and won't actually work as architected (because of your own bugs or because aws failover stuff has bugs and you can't even test it). Eg, falls over due to steep traffic spikes caused by outages when autoscaling mechanisms get previously unseen levels of load increases and enter some yoyo oscillation pattern, whole AZ is overloaded because all the failovers from the other failing AZ triggering at once, hit circuit breakers, spin up too slowly to ever pass health checks etc. Or can't detect something becoming glacially slow but not outright failing. See eg https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-down-outage-slack-imgur-hulu-asana-epic https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-d... & https://www.theverge.com/2020/11/25/21719396/amazon-web-services-aws-outage-down-internet https://www.theverge.com/2020/11/25/21719396/amazon-web-serv... etc (many more examples are out there)

If architected correctly, stuff deployed into aws stays up during all but the most extreme outages.

oars 4y ago

And then where do we discuss these extreme outages?

shawnz 4y ago
where did you discuss outages of HN's previous provider?
- dralley 4y ago
  Less people are interested in outages of HN's previous provider.
rubyist5eva 4y ago
reddit.... shudders
- mariojv 4y ago
  maybe not... https://aws.amazon.com/solutions/case-studies/reddit-aurora-case-study/ https://aws.amazon.com/solutions/case-studies/reddit-aurora-...
staticassertion 4y ago
If there's an extreme AWS outage, you're pretty much stuck to in-person or POTS.
VoidWhisperer 4y ago
HAM Radio
- namecheapTA 4y ago
  People joke but from the top of a decent mountain I can reach 45 miles into the bay area from me on a $150 2m radio in my truck with rooftop antenna.
  - loxias 4y ago
    That is so cool. If I had that (truck+radio) I'd be very tempted to scatter a few cheap relay nodes at good locations in the bay area. Monkeybrains+comcast is quite reliable.
  - iasay 4y ago
    The problem with that is the only people you have to talk to is hams. (Ex ham)
    - nwh5jg56df 4y ago
      Well same for computers & the internet :)
      - iasay 4y ago
        Fair point!
mekster 4y ago
Better to discuss the next day when it's over than seeing bunch of upset comments realtime posted.
- atmosx 4y ago
  If the AWS status wasn’t a static HTML page, I would agree.

fulafel 4y ago

Of those "correctly" architected apps, most are not properly tested for the failovers and won't actually work as architected (because of your own bugs or because aws failover stuff has bugs and you can't even test it). Eg, falls over due to steep traffic spikes caused by outages when autoscaling mechanisms get previously unseen levels of load increases and enter some yoyo oscillation pattern, whole AZ is overloaded because all the failovers from the other failing AZ triggering at once, hit circuit breakers, spin up too slowly to ever pass health checks etc. Or can't detect something becoming glacially slow but not outright failing. See eg https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-down-outage-slack-imgur-hulu-asana-epic https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-d... & https://www.theverge.com/2020/11/25/21719396/amazon-web-services-aws-outage-down-internet https://www.theverge.com/2020/11/25/21719396/amazon-web-serv... etc (many more examples are out there)