4 ms·

I hope this is only temporary. Where else will we discuss AWS outages when AWS goes down? Not even a joke.

by dralley 4y ago

cherioo 4y ago
The IP appears to be us-west-2, so we will still be able to discuss us-east-1 outages alright!
- ignoramous 4y ago
  but... the s3 buckets are in us-east-1, and postgres in ap-southeast-2. More regions better than one, for maximum impact with minimum effort.
  - pojzon 4y ago
    Is there at least one case when whole region went down ;)) ?
    - withinboredom 4y ago
      At least once, in 2012? I remember because just two days before we had fully switched over to AWS and hadn’t done multi-region yet. We got our first downtime in 10 years and there was nothing we could do, unlike when we had the servers in a colo. We were at the mercy of Amazon. After that, we moved everything back to real physical servers.
    - samspenc 4y ago
      Per the spreadsheet here https://awsmaniac.com/aws-outages/ https://awsmaniac.com/aws-outages/ : There seem to have been multiple "full" outages in 2011-12 in AWS' us-east-1 region, which, granted, is the oldest AWS region and likely has a bunch of legacy stuff. By "full" outages I mean that a few core services fell over but the entire region become inaccessible due to those core failures.
      - pojzon 4y ago
        Its over 10 years ago tho. Are there any RECENT full region outages ? Im forseeing a full downtime in Frankfurt this winter tho. Germany is in really bad position when it comes to electricity.
  - pmoriarty 4y ago
    The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service. Consider the extreme case where your service is scattered over every AWS region: here an outage of any AWS region is guaranteed to take down your service. Compare that to the case where your service is bound to only one region: then the odds of a single region outage taking down your entire service is reduced to 1 out of however many regions AWS has (assuming each region has an equal chance of suffering an outage). To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway).
    - ignoramous 4y ago
      > The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service. Agree. I think I should have suffixed a /s to my comment above. > To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway). Something, something... the greatest trick the devil (bigcloud) ever pulled...
f0e4c2f7 4y ago
If architected correctly, stuff deployed into aws stays up during all but the most extreme outages.
- oars 4y ago
  And then where do we discuss these extreme outages?
  - shawnz 4y ago
    where did you discuss outages of HN's previous provider?
    - dralley 4y ago
      Less people are interested in outages of HN's previous provider.
  - rubyist5eva 4y ago
    reddit.... shudders
    - mariojv 4y ago
      maybe not... https://aws.amazon.com/solutions/case-studies/reddit-aurora-case-study/ https://aws.amazon.com/solutions/case-studies/reddit-aurora-...
  - staticassertion 4y ago
    If there's an extreme AWS outage, you're pretty much stuck to in-person or POTS.
  - VoidWhisperer 4y ago
    HAM Radio
    - namecheapTA 4y ago
banana_giraffe 4y ago
It appears to be pointing at a bare EC2 instance, no doubt a lift-and-shift. Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic.
- pmoriarty 4y ago
  > Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic. You were just lucky enough not to have been affected by AWS outages, but many others were. You can get a lot of resilience to failure on AWS, but simply spinning up a dedicated EC2 instance is not nearly enough.
Pakdef 4y ago
Twitter is nice for learning about outages...
lazylion2 4y ago
/r/sysadmin

I hope this is only temporary. Where else will we discuss AWS outages when AWS goes down? Not even a joke.

cherioo 4y ago

The IP appears to be us-west-2, so we will still be able to discuss us-east-1 outages alright!

ignoramous 4y ago
but... the s3 buckets are in us-east-1, and postgres in ap-southeast-2. More regions better than one, for maximum impact with minimum effort.
- pojzon 4y ago
  Is there at least one case when whole region went down ;)) ?
  - withinboredom 4y ago
    At least once, in 2012? I remember because just two days before we had fully switched over to AWS and hadn’t done multi-region yet. We got our first downtime in 10 years and there was nothing we could do, unlike when we had the servers in a colo. We were at the mercy of Amazon. After that, we moved everything back to real physical servers.
  - samspenc 4y ago
    Per the spreadsheet here https://awsmaniac.com/aws-outages/ https://awsmaniac.com/aws-outages/ : There seem to have been multiple "full" outages in 2011-12 in AWS' us-east-1 region, which, granted, is the oldest AWS region and likely has a bunch of legacy stuff. By "full" outages I mean that a few core services fell over but the entire region become inaccessible due to those core failures.
    - pojzon 4y ago
      Its over 10 years ago tho. Are there any RECENT full region outages ? Im forseeing a full downtime in Frankfurt this winter tho. Germany is in really bad position when it comes to electricity.
- pmoriarty 4y ago
  The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service. Consider the extreme case where your service is scattered over every AWS region: here an outage of any AWS region is guaranteed to take down your service. Compare that to the case where your service is bound to only one region: then the odds of a single region outage taking down your entire service is reduced to 1 out of however many regions AWS has (assuming each region has an equal chance of suffering an outage). To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway).
  - ignoramous 4y ago
    > The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service. Agree. I think I should have suffixed a /s to my comment above. > To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway). Something, something... the greatest trick the devil (bigcloud) ever pulled...

f0e4c2f7 4y ago

If architected correctly, stuff deployed into aws stays up during all but the most extreme outages.

oars 4y ago
And then where do we discuss these extreme outages?
- shawnz 4y ago
  where did you discuss outages of HN's previous provider?
  - dralley 4y ago
    Less people are interested in outages of HN's previous provider.
- rubyist5eva 4y ago
  reddit.... shudders
  - mariojv 4y ago
    maybe not... https://aws.amazon.com/solutions/case-studies/reddit-aurora-case-study/ https://aws.amazon.com/solutions/case-studies/reddit-aurora-...
- staticassertion 4y ago
  If there's an extreme AWS outage, you're pretty much stuck to in-person or POTS.
- VoidWhisperer 4y ago
  HAM Radio
  - namecheapTA 4y ago

banana_giraffe 4y ago

It appears to be pointing at a bare EC2 instance, no doubt a lift-and-shift. Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic.

pmoriarty 4y ago
> Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic. You were just lucky enough not to have been affected by AWS outages, but many others were. You can get a lot of resilience to failure on AWS, but simply spinning up a dedicated EC2 instance is not nearly enough.

Pakdef 4y ago

Twitter is nice for learning about outages...

lazylion2 4y ago

/r/sysadmin