5 ms·

Call it the aws holiday. Most other companies will be down anyway. It's very likely that your company can afford to be down for a few hours, too.

chii 1y ago
imagine if the electricity supplier too that stance.
- ahoka 1y ago
  Isn't that basically Texas?
  - SecretDreams 1y ago
    Texas is like if you ran your cloud entirely in SharePoint.
    - jofzar 1y ago
      Let's not insult SharePoint like that. It's like if you ran you cloud on an old dell box in your closet while your parent company is offering to directly host it in AWS for free.
      - sgarland 1y ago
        Also, every time your cloud went down, the parent company begged you to reconsider, explaining that all they need you to do is remove the disturbingly large cobwebs so they can migrate it. You tell them that to do so would violate your strongly-held beliefs, and when they stare at you in bewilderment, you yell “FREEDOM!” while rolling armadillos at them like they’re bowling balls.
- malfist 1y ago
  But that is the stance for a lot of electrical utilities. Sometimes weather or a car wreck takes out power and since its too expensive to have spares everywhere, sometimes you have to wait a few hours for a spare to be brought in
  - yuliyp 1y ago
    No, that's not the stance for electrical utilities (at least in most developed countries, including the US): the vast majority of weather events cause localized outages (the grid as a whole has redundancies built in; distribution to (residential and some industrial) does not. It expects failures of some power plants, transmission lines, etc. and can adapt with reserve power, or, in very rare cases by partial degradation (i.e. rolling blackouts). It doesn't go down fully.
    - JoyfulTurkey 1y ago
      Spain and Portugal had a massive power outage this spring, no?
      - formerly_proven 1y ago
        Yeah, and it has a 30 page Wikipedia article with 161 sources (https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackout https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...). Does that seem like a common occurrence?
  - crote 1y ago
    > Sometimes weather or a car wreck takes out power Not really? Most of the infrastructure is quite resilient and the rare outage is usually limited to a street or two, with restoration time mainly determined by the time it takes the electricians to reach the incident site. For any given address that's maybe a few hours per decade - with the most likely cause being planned maintenance. That's not a "spares are too expensive" issue, that's a "giving every home two fully independent power feeds is silly" issue. Anything on a metro-sized level is pretty much unheard of, and will be treated as serious as a plane crash. They can essentially only be caused by systemic failure on multiple levels, as the grid is configured to survive multiple independent failures at the same time. Comparing that to the AWS world: individual servers going down is inevitable and shouldn't come as a surprise. Everyone has redundancies, and an engineer accidentally yanking the power cables of an entire rack shouldn't even be noticeable to any customers. But an entire service going down across an entire availability zone? That should be virtually impossible, and having it happen regularly is a bit of a red flag.
- awillen 1y ago
  That's the wrong analogy though. We're not talking about the supplier - I'm sure Amazon is doing its damnedest to make sure that AWS isn't going down. The right analogy is to imagine if businesses that used electricity took that stance, and they basically all do. If you're a hospital or some other business where a power outage is life or death, you plan by having backup generators. But if you're the overwhelming majority of businesses, you do absolutely nothing to ensure that you have power during a power outage, and it's fine.
  - chii 1y ago
    > But if you're the overwhelming majority of businesses, you do absolutely nothing to ensure that you have power during a power outage, and it's fine. it is fine because the electricity supplier is so good today that people don't see it going down as a risk. Look at south africa's electricity supplier for a different scenario.
- hnlmorg 1y ago
  Utility companies do not have redundancy for every part of their infrastructure either. Hence why severe weather or other unexpected failures can cause loss of power, internet or even running water. Texas has had statewide power outages. Spain and Portugal suffered near-nationwide power outages last year. Many US states are heavily reliant on the same single source for water. And remember the discussions on here about Europe's reliance on Russian gas? Then you have the XKCD sketch about how most software products are reliant on at least one piece of open source software that is maintained by a single person as a hobby. Nobody likes a single point of failure but often the costs associated with mitigating that are much greater than the risks of having that point of failure. This is why "risk assessments" are a thing.
  - pyrale 1y ago
    > Hence why severe weather or other unexpected failures can cause loss of power, internet or even running water. Not all utility companies have the same policies, but all have a resiliency plan to avoid blackout that is a bit more serious than "Just run it on AWS".
    - hnlmorg 1y ago
      > Not all utility companies have the same policies, but all have a resiliency plan to avoid blackout that is a bit more serious than "Just run it on AWS". You're arguing as if "run it on AWS" was a decision that didn't undergo the same kinds of risk assessment. As someone who's had to complete such processes (and in some companies, even define them), I can assure you that nobody of any competency runs stuff on AWS complacently. In fact running stuff with resilience in AWS isn't even as simple as "just running it in AWS". There's a whole plethora of things to consider, and each with its own costs attached. As the meme goes "one does not simply just run something on AWS"
      - pyrale 1y ago
        > nobody of any competency runs stuff on AWS complacently. I agree with this. My point is simply that we, as an industry, are not a very competent bunch when it comes to risk management ; and that's especially true when compared to TSOs. That doesn't mean nobody knows what they do in our industry or that shit never hits the fan elsewhere, but I would argue that it's an outlier behaviour, whereas it's the norm in more secure industries. > As the meme goes "one does not simply just run something on AWS" The meme has currency for a reason, unfortunately. --- That being said, my original point was that utilities losing clients after a storm isn't the consequence of bad (or no) risk assessment ; it's the consequence of them setting up acceptable loss thresholds depending on the likelihood of an event happening, and making sure that the network as a whole can respect these SLOs while strictly respecting safety criteria.
- throw0101d 1y ago
  > imagine if the electricity supplier too that stance. Imagine if the cloud supplier was actually as important as the electricity supplier. But since you mention it, there are instances of this and provisions for getting back up and running: * https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackout https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou... * https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
  - chii 1y ago
    and how many times have aws gone down majorly like that? I think you wouldn't be able to count it.
    - throw0101d 1y ago
      * https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Services#Outages https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service... As someone who lives in Ontario, Canada, I got hit by the 2003 grid outage, which is once in >20 years. Seems like a fairly good uptime to me. (Each electrical grid can perhaps be considered analogous to a separate cloud provider. Or perhaps, in US/CA, regions: * https://en.wikipedia.org/wiki/North_American_Electric_Reliability_Corporation https://en.wikipedia.org/wiki/North_American_Electric_Reliab... )
- DiggyJohnson 1y ago
  The electric grid is much more important than most private sector software projects by an order of magnitude. Catastrophic data loss or lack of disaster recovery kills companies. AWS outages do not.
  - umeshunni 1y ago
    What if the electricity grid depends on some AWS service?
    - quaintdev 1y ago
      That would be circular dependency.
      - croes 1y ago
        And?
      - AnIrishDuck 1y ago
        The grid actually already has a fair number of (non-software) circular dependencies. This is why they have black start [1] procedures and run drills of those procedures. Or should, at least; there have been high profile outages recently that have exposed holes in these plans [2]. 1. https://en.wikipedia.org/wiki/Black_start https://en.wikipedia.org/wiki/Black_start 2. https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackout#Power_restoration https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...
    - thatguy0900 1y ago
      It doesn't though? Weird what if
      - jacquesm 1y ago
        You'd be surprised. See. GP asks a very interesting question. And some grid infra indeed relies on AWS, definitely not all of it but there are some aspects of it that are hosted by AWS.
- kelnos 1y ago
  Fortunately nearly all services running on AWS aren't as important as the electric utility, so this argument is not particularly relevant. And regardless, electric service all over the world goes down for minutes or hours all the time.

chii 1y ago

imagine if the electricity supplier too that stance.

ahoka 1y ago
Isn't that basically Texas?
- SecretDreams 1y ago
  Texas is like if you ran your cloud entirely in SharePoint.
  - jofzar 1y ago
    Let's not insult SharePoint like that. It's like if you ran you cloud on an old dell box in your closet while your parent company is offering to directly host it in AWS for free.
    - sgarland 1y ago
      Also, every time your cloud went down, the parent company begged you to reconsider, explaining that all they need you to do is remove the disturbingly large cobwebs so they can migrate it. You tell them that to do so would violate your strongly-held beliefs, and when they stare at you in bewilderment, you yell “FREEDOM!” while rolling armadillos at them like they’re bowling balls.
malfist 1y ago
But that is the stance for a lot of electrical utilities. Sometimes weather or a car wreck takes out power and since its too expensive to have spares everywhere, sometimes you have to wait a few hours for a spare to be brought in
- yuliyp 1y ago
  No, that's not the stance for electrical utilities (at least in most developed countries, including the US): the vast majority of weather events cause localized outages (the grid as a whole has redundancies built in; distribution to (residential and some industrial) does not. It expects failures of some power plants, transmission lines, etc. and can adapt with reserve power, or, in very rare cases by partial degradation (i.e. rolling blackouts). It doesn't go down fully.
  - JoyfulTurkey 1y ago
    Spain and Portugal had a massive power outage this spring, no?
    - formerly_proven 1y ago
      Yeah, and it has a 30 page Wikipedia article with 161 sources (https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackout https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...). Does that seem like a common occurrence?
- crote 1y ago
  > Sometimes weather or a car wreck takes out power Not really? Most of the infrastructure is quite resilient and the rare outage is usually limited to a street or two, with restoration time mainly determined by the time it takes the electricians to reach the incident site. For any given address that's maybe a few hours per decade - with the most likely cause being planned maintenance. That's not a "spares are too expensive" issue, that's a "giving every home two fully independent power feeds is silly" issue. Anything on a metro-sized level is pretty much unheard of, and will be treated as serious as a plane crash. They can essentially only be caused by systemic failure on multiple levels, as the grid is configured to survive multiple independent failures at the same time. Comparing that to the AWS world: individual servers going down is inevitable and shouldn't come as a surprise. Everyone has redundancies, and an engineer accidentally yanking the power cables of an entire rack shouldn't even be noticeable to any customers. But an entire service going down across an entire availability zone? That should be virtually impossible, and having it happen regularly is a bit of a red flag.
awillen 1y ago
That's the wrong analogy though. We're not talking about the supplier - I'm sure Amazon is doing its damnedest to make sure that AWS isn't going down. The right analogy is to imagine if businesses that used electricity took that stance, and they basically all do. If you're a hospital or some other business where a power outage is life or death, you plan by having backup generators. But if you're the overwhelming majority of businesses, you do absolutely nothing to ensure that you have power during a power outage, and it's fine.
- chii 1y ago
  > But if you're the overwhelming majority of businesses, you do absolutely nothing to ensure that you have power during a power outage, and it's fine. it is fine because the electricity supplier is so good today that people don't see it going down as a risk. Look at south africa's electricity supplier for a different scenario.
hnlmorg 1y ago
Utility companies do not have redundancy for every part of their infrastructure either. Hence why severe weather or other unexpected failures can cause loss of power, internet or even running water. Texas has had statewide power outages. Spain and Portugal suffered near-nationwide power outages last year. Many US states are heavily reliant on the same single source for water. And remember the discussions on here about Europe's reliance on Russian gas? Then you have the XKCD sketch about how most software products are reliant on at least one piece of open source software that is maintained by a single person as a hobby. Nobody likes a single point of failure but often the costs associated with mitigating that are much greater than the risks of having that point of failure. This is why "risk assessments" are a thing.
- pyrale 1y ago
  > Hence why severe weather or other unexpected failures can cause loss of power, internet or even running water. Not all utility companies have the same policies, but all have a resiliency plan to avoid blackout that is a bit more serious than "Just run it on AWS".
  - hnlmorg 1y ago
    > Not all utility companies have the same policies, but all have a resiliency plan to avoid blackout that is a bit more serious than "Just run it on AWS". You're arguing as if "run it on AWS" was a decision that didn't undergo the same kinds of risk assessment. As someone who's had to complete such processes (and in some companies, even define them), I can assure you that nobody of any competency runs stuff on AWS complacently. In fact running stuff with resilience in AWS isn't even as simple as "just running it in AWS". There's a whole plethora of things to consider, and each with its own costs attached. As the meme goes "one does not simply just run something on AWS"
    - pyrale 1y ago
      > nobody of any competency runs stuff on AWS complacently. I agree with this. My point is simply that we, as an industry, are not a very competent bunch when it comes to risk management ; and that's especially true when compared to TSOs. That doesn't mean nobody knows what they do in our industry or that shit never hits the fan elsewhere, but I would argue that it's an outlier behaviour, whereas it's the norm in more secure industries. > As the meme goes "one does not simply just run something on AWS" The meme has currency for a reason, unfortunately. --- That being said, my original point was that utilities losing clients after a storm isn't the consequence of bad (or no) risk assessment ; it's the consequence of them setting up acceptable loss thresholds depending on the likelihood of an event happening, and making sure that the network as a whole can respect these SLOs while strictly respecting safety criteria.
throw0101d 1y ago
> imagine if the electricity supplier too that stance. Imagine if the cloud supplier was actually as important as the electricity supplier. But since you mention it, there are instances of this and provisions for getting back up and running: * https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackout https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou... * https://en.wikipedia.org/wiki/Northeast_blackout_of_2003 https://en.wikipedia.org/wiki/Northeast_blackout_of_2003
- chii 1y ago
  and how many times have aws gone down majorly like that? I think you wouldn't be able to count it.
  - throw0101d 1y ago
    * https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Services#Outages https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service... As someone who lives in Ontario, Canada, I got hit by the 2003 grid outage, which is once in >20 years. Seems like a fairly good uptime to me. (Each electrical grid can perhaps be considered analogous to a separate cloud provider. Or perhaps, in US/CA, regions: * https://en.wikipedia.org/wiki/North_American_Electric_Reliability_Corporation https://en.wikipedia.org/wiki/North_American_Electric_Reliab... )
DiggyJohnson 1y ago
The electric grid is much more important than most private sector software projects by an order of magnitude. Catastrophic data loss or lack of disaster recovery kills companies. AWS outages do not.
- umeshunni 1y ago
  What if the electricity grid depends on some AWS service?
  - quaintdev 1y ago
    That would be circular dependency.
    - croes 1y ago
      And?
    - AnIrishDuck 1y ago
      The grid actually already has a fair number of (non-software) circular dependencies. This is why they have black start [1] procedures and run drills of those procedures. Or should, at least; there have been high profile outages recently that have exposed holes in these plans [2]. 1. https://en.wikipedia.org/wiki/Black_start https://en.wikipedia.org/wiki/Black_start 2. https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackout#Power_restoration https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...
  - thatguy0900 1y ago
    It doesn't though? Weird what if
    - jacquesm 1y ago
      You'd be surprised. See. GP asks a very interesting question. And some grid infra indeed relies on AWS, definitely not all of it but there are some aspects of it that are hosted by AWS.
kelnos 1y ago
Fortunately nearly all services running on AWS aren't as important as the electric utility, so this argument is not particularly relevant. And regardless, electric service all over the world goes down for minutes or hours all the time.