3 ms·

They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the

They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook.

palmotea 1y ago
>> People laughed and some said something like "well, if something like AWS falls then we have bigger problems". > They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook. They made their own bigger problems by all crowding into the same single region.
- pluto_modadic 1y ago
  it's a weird effect: imagine a beach, with icecream vendors. You'd think it would be optimal for two vendors to each split it half north, half south. However, in wanting to steal some of the other vendors' customers, you end up with two icecream stands in the center. So too with outages. Safety / loss of blame in numbers.
baobabKoodaa 1y ago
Plenty of stuff still works.
- swat535 1y ago
  Sure but to their point, you're off the hook if half of the internet is down.. it's sort of like: "No one gets fired for picking IBM".
  - BenjiWiebe 1y ago
    I'd rather my services and products be in the half of the Internet that works than the half that doesn't.
    - Imustaskforhelp 1y ago
      I feel like I don't really like AWS and prefer self hosted vps or even google cloud / cloudflare more but and so I agree with what you are trying to say but Let me be the devil's argument. I mean I agree but what you are saying is that where else are you gonna host it? If you host it yourself and then it turns out to be an issue and you go down then that's entirely on you and 99% of the internet still works. But if Aws goes down, lets say 50% of the internet goes down. So, in essence, nobody blames a particular team/person just as the parent comment said that nobody gets fired for picking IBM. Although, I still think that the idea which is worrying is such massive centralization of servers that we have a single switch which can turn half the internet off. So I am a bit worried from the centralization side of thing's.
      - BenjiWiebe 1y ago
        It was pretty frustrating to me when multiple services weren't working during the outage. I'm an end user; I don't use AWS, nor do I work on anything using AWS. From my perspective, multiple unrelated websites quit working at the same time. I would rather have had one website down, and the rest working, than for me to be completely hamstrung because so many services are down simultaneously.
oceanplexian 1y ago
Why would your competitors go down? AWS has at best 30-35% market share. And that's ignoring the huge mass of companies who still run their infrastructure on bare metal.
- codeduck 1y ago
  because your competitors are probably using services that depend on AWS.
- CharlieDigital 1y ago
  A whole bunch of meeting bots use Recall. Recall is on AWS. Everyone using Recall for meeting recordings is down. In some domains, a single SaaS dominates the domain and if that SaaS sits on AWS, it doesn't matter if AWS is 35% marketshare because the SaaS that dominates 80% of the domain is on AWS so the effect is wider than just AWS's market share. We're on GCP, but we have various SaaS vendors on AWS so any of the services that rely on AWS are gone. Many chat/meeting services also run on AWS Chime so even if you're not on AWS, if a vendor uses Chime, that service is down.
  - senderista 1y ago
    Doesn’t Slack use Chime for video calls?
    - CharlieDigital 1y ago
      Yes. Yes it does.
- tetha 1y ago
  Part of the company I work at is doing infrastructure consulting. We're in fact seeing companies moving to bare metal, with the rise of turnkey container systems between Nutanix, Purestorage, Redhat, ... At this point in time, a few remotely managed boxes in a rack can offer a really good experience for containers for very little effort. And this comes in a time with regulations like Dora and the BaFin tightening things - managing these boxes becomes less effort than maintaining compliance across vendors.
immibis 1y ago
Our app was up. I'm sure we made a lot of money.
- joshstrange 1y ago
  The question really becomes, did you make money that you wouldn't have made when services came back up? As in, will people just shift their purchase time to tomorrow when you are back online? Sure, some % is completely lost but you have to weigh that lost amount against the ongoing costs to be multi-cloud (or multi-provider) and the development time against those costs. For most people I think it's cheaper to just be down for a few hours. Yes, this outage is longer than any I can remember but most people will shrug it off and move on once it comes back up fully. At the end of the day most of us aren't working on super critical things. No one is dying because they can't purchase X item online or use Y SaaS. And, more importantly, customers are _not_ willing to pay the extra for you to host your backend in multiple regions/providers. In my contracts (for my personal company) I call out the single-point-of-failure very clearly and I've never had anyone balk. If they did I'd offer then resiliency (for a price) and I have no doubt that they would opt to "roll the dice" instead of pay. Lastly, it's near-impossible to verify what all your vendors are using so even if you manage to get everything resilient it only takes one chink in the armor the bring it all down (See: us-east-1 and various AWS services that rely on that even if you don't host anything in us-east-1 directly). I'm not trying to downplay this, pretend it doesn't matter, or anything like that. Just trying to point out that most people don't care because no one seems to care (or want to pay for it). I wish that was different (I wish a lot of things were different) but wishing doesn't pay my bills and so if customers don't want to pay for resiliency then this is what they get and I'm at peace with that.

They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook.

palmotea 1y ago

>> People laughed and some said something like "well, if something like AWS falls then we have bigger problems". > They’re not wrong though. If AWS goes down, EVERYTHING goes down to some degree. Your app, your competitor’s apps, your clients’ chat apps. You’re kinda off the hook. They made their own bigger problems by all crowding into the same single region.

pluto_modadic 1y ago
it's a weird effect: imagine a beach, with icecream vendors. You'd think it would be optimal for two vendors to each split it half north, half south. However, in wanting to steal some of the other vendors' customers, you end up with two icecream stands in the center. So too with outages. Safety / loss of blame in numbers.

baobabKoodaa 1y ago

Plenty of stuff still works.

swat535 1y ago
Sure but to their point, you're off the hook if half of the internet is down.. it's sort of like: "No one gets fired for picking IBM".
- BenjiWiebe 1y ago
  I'd rather my services and products be in the half of the Internet that works than the half that doesn't.
  - Imustaskforhelp 1y ago
    I feel like I don't really like AWS and prefer self hosted vps or even google cloud / cloudflare more but and so I agree with what you are trying to say but Let me be the devil's argument. I mean I agree but what you are saying is that where else are you gonna host it? If you host it yourself and then it turns out to be an issue and you go down then that's entirely on you and 99% of the internet still works. But if Aws goes down, lets say 50% of the internet goes down. So, in essence, nobody blames a particular team/person just as the parent comment said that nobody gets fired for picking IBM. Although, I still think that the idea which is worrying is such massive centralization of servers that we have a single switch which can turn half the internet off. So I am a bit worried from the centralization side of thing's.
    - BenjiWiebe 1y ago
      It was pretty frustrating to me when multiple services weren't working during the outage. I'm an end user; I don't use AWS, nor do I work on anything using AWS. From my perspective, multiple unrelated websites quit working at the same time. I would rather have had one website down, and the rest working, than for me to be completely hamstrung because so many services are down simultaneously.

oceanplexian 1y ago

Why would your competitors go down? AWS has at best 30-35% market share. And that's ignoring the huge mass of companies who still run their infrastructure on bare metal.

codeduck 1y ago
because your competitors are probably using services that depend on AWS.
CharlieDigital 1y ago
A whole bunch of meeting bots use Recall. Recall is on AWS. Everyone using Recall for meeting recordings is down. In some domains, a single SaaS dominates the domain and if that SaaS sits on AWS, it doesn't matter if AWS is 35% marketshare because the SaaS that dominates 80% of the domain is on AWS so the effect is wider than just AWS's market share. We're on GCP, but we have various SaaS vendors on AWS so any of the services that rely on AWS are gone. Many chat/meeting services also run on AWS Chime so even if you're not on AWS, if a vendor uses Chime, that service is down.
- senderista 1y ago
  Doesn’t Slack use Chime for video calls?
  - CharlieDigital 1y ago
    Yes. Yes it does.
tetha 1y ago
Part of the company I work at is doing infrastructure consulting. We're in fact seeing companies moving to bare metal, with the rise of turnkey container systems between Nutanix, Purestorage, Redhat, ... At this point in time, a few remotely managed boxes in a rack can offer a really good experience for containers for very little effort. And this comes in a time with regulations like Dora and the BaFin tightening things - managing these boxes becomes less effort than maintaining compliance across vendors.

immibis 1y ago

Our app was up. I'm sure we made a lot of money.

joshstrange 1y ago
The question really becomes, did you make money that you wouldn't have made when services came back up? As in, will people just shift their purchase time to tomorrow when you are back online? Sure, some % is completely lost but you have to weigh that lost amount against the ongoing costs to be multi-cloud (or multi-provider) and the development time against those costs. For most people I think it's cheaper to just be down for a few hours. Yes, this outage is longer than any I can remember but most people will shrug it off and move on once it comes back up fully. At the end of the day most of us aren't working on super critical things. No one is dying because they can't purchase X item online or use Y SaaS. And, more importantly, customers are _not_ willing to pay the extra for you to host your backend in multiple regions/providers. In my contracts (for my personal company) I call out the single-point-of-failure very clearly and I've never had anyone balk. If they did I'd offer then resiliency (for a price) and I have no doubt that they would opt to "roll the dice" instead of pay. Lastly, it's near-impossible to verify what all your vendors are using so even if you manage to get everything resilient it only takes one chink in the armor the bring it all down (See: us-east-1 and various AWS services that rely on that even if you don't host anything in us-east-1 directly). I'm not trying to downplay this, pretend it doesn't matter, or anything like that. Just trying to point out that most people don't care because no one seems to care (or want to pay for it). I wish that was different (I wish a lot of things were different) but wishing doesn't pay my bills and so if customers don't want to pay for resiliency then this is what they get and I'm at peace with that.