5 ms·

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a f

Whenever us-east-1 goes down you just get a really good feel for how many other companies also have pretty fragile setups. They apps I work on can deal with a few hours of downtime, so as long as I'm sure I can recover from getting totally leveled its ok. And I think it's that was for the majority of companies. Most don't want the extra effort and cost of failover.

notfromhere 5y ago
Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies
- jaxrtech 5y ago
  > Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies Was it Metra or the CTA out of curiosity?
  - phantomread 5y ago
    In Chicago I believe “commuter rail” almost always refers to Metra.
- kennydude 5y ago
  Why is a commuter rail travel system dependent on AWS? O___O
  - t3rabytes 5y ago
    They decided to run their infra in AWS instead of another cloud provider or on-prem? why is McDonalds dependent on AWS for their app to work? (/s)
    - joe_chip 5y ago
      Did you just equivocate a public transit service with McDonald's?
  - wil421 5y ago
    From what I’ve seen about Government IT it’s probably a good thing it’s on AWS. Why shouldn’t Government IT be using the same tools regular companies use for IT?
    - kennydude 5y ago
      Government IT using AWS is fine. Using it for safety critical things like railways is a lot more of an issue. What if there's any kind of latency etc?
throwntoday 5y ago
I'm just surprised Amazon was not using something sectioned off from the rest of AWS.
- anamexis 5y ago
  Possibly stems from one of the tenets of the apocryphal Bezos decree that started AWS[1] > All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. 1: https://nordicapis.com/the-bezos-api-mandate-amazons-manifesto-for-externalization/ https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...
  - hef19898 5y ago
    That attitude, and the ability to enforce it at the scale Amazon is at, are probably the biggest strength of Bezos. It does cover all aspects so, including work culture...
- matteotom 5y ago
  My understanding is there's a pretty strong culture of running amazon.com directly in AWS as much as possible. I don't think there's much reason to believe a sectioned off alternative would be much more reliable, it would just break at different times maybe.
- deleted 5y ago
  [deleted]
nemothekid 5y ago
I'm still surprised how many people are still using us-east-1.
errcorrectcode 5y ago
The frightening thing is how fragile it seems. If you're going to concentrate risk on AWS, it better be essentially flawless, stable, and highly-redundant.
- nexuist 5y ago
  It is, most of the time. That's why these outages are such a big deal. Nobody would care if it went out multiple times a month.
professoretc 5y ago
It's finals season, so all the students taking online exams on Canvas had a fun day.
peterkelly 5y ago
I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is statistically likely to happen at some point, so ignoring it is setting yourself up for major (and potentially very expensive) problems if you don't test for it regularly.
- kall 5y ago
  At the project level maybe. At the AWS level, that‘s ridiculous. A lot of services can live with one major outage a year, but not once a month. Completely failing when their region goes down is reasonable.
  - deleted 5y ago
    [deleted]
  - peterkelly 5y ago
    Region failures can be non-recoverable (e.g. natural disaster). One major outage per year doesn't necessarily mean just a few hours of downtime, it could mean having to redeploy your entire service somewhere else, which could take several days or more if you haven't prepared for it. How many of those services can live with that?
    - stdbrouw 5y ago
      > How many of those services can live with that? Apparently, many can! Look at how many companies choose not to pay ransom when hit with a ransomware attack, or prefer to negotiate for days instead of buckling straight away, even if it means operations are crippled for weeks. They don't typically go bankrupt afterwards, everyone coped, life moves on.
- ec109685 5y ago
  Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html http://techblog.netflix.com/2013/12/active-active-for-multi-...

notfromhere 5y ago

Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies

jaxrtech 5y ago
> Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies Was it Metra or the CTA out of curiosity?
- phantomread 5y ago
  In Chicago I believe “commuter rail” almost always refers to Metra.
kennydude 5y ago
Why is a commuter rail travel system dependent on AWS? O___O
- t3rabytes 5y ago
  They decided to run their infra in AWS instead of another cloud provider or on-prem? why is McDonalds dependent on AWS for their app to work? (/s)
  - joe_chip 5y ago
    Did you just equivocate a public transit service with McDonald's?
- wil421 5y ago
  From what I’ve seen about Government IT it’s probably a good thing it’s on AWS. Why shouldn’t Government IT be using the same tools regular companies use for IT?
  - kennydude 5y ago
    Government IT using AWS is fine. Using it for safety critical things like railways is a lot more of an issue. What if there's any kind of latency etc?

throwntoday 5y ago

I'm just surprised Amazon was not using something sectioned off from the rest of AWS.

anamexis 5y ago
Possibly stems from one of the tenets of the apocryphal Bezos decree that started AWS[1] > All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. 1: https://nordicapis.com/the-bezos-api-mandate-amazons-manifesto-for-externalization/ https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...
- hef19898 5y ago
  That attitude, and the ability to enforce it at the scale Amazon is at, are probably the biggest strength of Bezos. It does cover all aspects so, including work culture...
matteotom 5y ago
My understanding is there's a pretty strong culture of running amazon.com directly in AWS as much as possible. I don't think there's much reason to believe a sectioned off alternative would be much more reliable, it would just break at different times maybe.
deleted 5y ago
[deleted]

nemothekid 5y ago

I'm still surprised how many people are still using us-east-1.

errcorrectcode 5y ago

The frightening thing is how fragile it seems. If you're going to concentrate risk on AWS, it better be essentially flawless, stable, and highly-redundant.

nexuist 5y ago
It is, most of the time. That's why these outages are such a big deal. Nobody would care if it went out multiple times a month.

professoretc 5y ago

It's finals season, so all the students taking online exams on Canvas had a fun day.

peterkelly 5y ago

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is statistically likely to happen at some point, so ignoring it is setting yourself up for major (and potentially very expensive) problems if you don't test for it regularly.

kall 5y ago
At the project level maybe. At the AWS level, that‘s ridiculous. A lot of services can live with one major outage a year, but not once a month. Completely failing when their region goes down is reasonable.
- deleted 5y ago
  [deleted]
- peterkelly 5y ago
  Region failures can be non-recoverable (e.g. natural disaster). One major outage per year doesn't necessarily mean just a few hours of downtime, it could mean having to redeploy your entire service somewhere else, which could take several days or more if you haven't prepared for it. How many of those services can live with that?
  - stdbrouw 5y ago
    > How many of those services can live with that? Apparently, many can! Look at how many companies choose not to pay ransom when hit with a ransomware attack, or prefer to negotiate for days instead of buckling straight away, even if it means operations are crippled for weeks. They don't typically go bankrupt afterwards, everyone coped, life moves on.
ec109685 5y ago
Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html http://techblog.netflix.com/2013/12/active-active-for-multi-...