7 ms·

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have thi

Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically eroding that strength.

hvb2 1y ago
> The internet got its main strengths from the fact that it was completely decentralized. Decentralized in terms of many companies making up the internet. Yes we've seen heavy consolidation in now having less than 10 companies make up the bulk of the internet. The problem here isn't caused by companies chosing one cloud provider over the other. It's the economies of scale leading us to few large companies in any sector.
- jacquesm 1y ago
  I think one reason is that people are just bad at statistics. Chance of materialization * impact = small. Sure. Over a short enough time that's true for any kind of risk. But companies tend to live for years, decades even and sometimes longer than that. If we're going to put all of those precious eggs in one basket, as long as the basket is substantially stronger than the eggs we're fine, right? Until the day someone drops the basket. And over a long enough time span all risks eventually materialize. So we're playing this game, and usually we come out ahead. But trust me, 10 seconds after this outage is solved everybody will have forgotten about the possibility.
  - hvb2 1y ago
    Absolutely, but the cost of perfection (100% uptime in this case) is infinite. As long as the outages are rare enough and you automatically fail over to a different region, what's the problem?
    - jacquesm 1y ago
      Often simply the lack of a backup outside of the main cloud account.
      - hvb2 1y ago
        Sure, but on a typical outage how likely is it that you'll have that all up and running before the outage is resolved? And secondly, how often do you create that backup and are you willing to lose the writes since the last backup? That backup is absolutely something people should have, but I doubt those are ever used to bring a service back up. That would be a monumental failure of your hosting provider (colo/cloud/whatever)
        jacquesm 1y ago
        > Sure, but on a typical outage how likely is it that you'll have that all up and running before the outage is resolved? Not, but if some Amazon flunky decides to kill your account to protect the Amazon brand then you will at least survive, even if you'll lose some data.
- lentil_soup 1y ago
  > Decentralized in terms of many companies making up the internet Not companies, the protocols are decentralized and at some point it was mostly non-companies. Anyone can hook up a computer and start serving requests which was/is a radical concept, we've lost a lot, unfortunately
  - hvb2 1y ago
    No we've not lost that at all. Nobody prevents you from doing that. We have put more and more services on fewer and fewer vendors. But that's the consolidation and cost point.
    - IlikeKitties 1y ago
      > No we've not lost that at all. Nobody prevents you from doing that. May I introduce you to our Lord and Slavemaster CGNAT?
      - yupyupyups 1y ago
        That depends on who your ISP is.
      - otterley 1y ago
        There’s more than one way to get a server on the Internet. You can pay a local data center to put your machine in one of their racks.
      - tcfhgj 1y ago
        may I introduce you to ipv6?
        IlikeKitties 1y ago
        Try hosting an E-Mail Server behind CGNat and send an E-Mail to...anyone really.
        1y ago
- psychoslave 1y ago
  Well, that is exactly what resilient distributed network are about. Not that much the technical details we implement them through, but the social relationship and balanced in political decision power. Be it a company or a state, concentration of power that exeeds the needs for their purpose to function by a large margin is always a sure way to spread corruption, create feedback loop in single point of failure, and is buying everyone a ticket to some dystopian reality with a level of certainty that beats anything that a SLA will ever give us.
- chasd00 1y ago
  Decentralized with respect to connectivity. If a construction crew cuts a fiber bundle routing protocols will route around the damage and packets keep showing up at the destination. Or, only a localized group of users will be affected. That level of decentralization is not what we have at higher levels in the stack with AWS being a good example. Even connectivity has it's points of failure. I've touched with my own hands fiber runs that, with a few quick snips from a wire cutter, could bring sizable portions of the Internet offline. Granted that was a long time ago so those points of failure may no longer exist.
padjo 1y ago
Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.
- jacquesm 1y ago
  Thank you for illustrating my point. You didn't even bother to read the second paragraph.
  - shawabawa3 1y ago
    > Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically eroding that strength. my business contingency plan for "AWS shuts down and never comes back up" is to go bankrupt
    - jacquesm 1y ago
      Is that also your contingency plan for 'user uploads objectionable content and alerts Amazon to get your account shut down'? Make sure you let your investors know.
      - padjo 1y ago
        If your mitigation for that risk is to have an elaborate plan to move to a different cloud provider, where the same problem can just happen again, then you’re doing an awful job of risk management.
        jacquesm 1y ago
        > If your mitigation for that risk is to have an elaborate plan to move to a different cloud provider, where the same problem can just happen again, then you’re doing an awful job of risk management. Where did I say that? If I didn't say it: could you please argue in good faith. Thank you.
raincole 1y ago
Most companies just aren't important enough to worry about "AWS never come back up." Planning for this case is just like planning for a terrorist blowing up your entire office. If you're the Pentagon sure you'd better have a plan for that. But most companies are not the Pentagon.
- Frieren 1y ago
  > Most companies just aren't important enough to worry about "AWS never come back up." But a large enough of "not too big to fail" companies become a too big to fail event. Too many medium sized companies have a total dependency on AWS or Google or Microsoft, if not several at the same time. We live in a increasingly fragile society, one step closer to critical failure because big tech is not regulated in the same way than other infrastructure.
  - raincole 1y ago
    Well I agree. I kinda think the AI apocalypse would not be like a skynet killing us, but a malware be patched onto all the Tesla and causes one million crashes tomorrow morning.
    - swader999 1y ago
      Battery fires.
  - coffeebeqn 1y ago
    Many have a hard dependency on AWS && Google && Microsoft!
- paulddraper 1y ago
  Exactly. And FWIW, "AWS is down"....only one region (out of 36) of AWS is down. You can do the multi-region failover, though that's still possibly overkill for most.
anal_reactor 1y ago
First, planning for AWS outage is pointless. Unless you provide service of national security or something, your customers are going to understand that when there's global internet outage your service doesn't work either. The cost of maintaining a working failover across multiple cloud providers is just too high compared to potential benefits. It's astonishing that so few eningeers understand the fact that maintaining a technically beautiful solution costs time and money, which might not make a justified business case. Second, preparing for the disappearance of AWS is even more silly. The chance that it will happen are orders of magnitude smaller than the chance that the cost of preparing for such an event will kill your business. Let me ask you: how do you prepare your website for the complete collapse of western society? Will you be able to adapt your business model to post-apocalyptic world where it's only cockroaches?
- jacquesm 1y ago
  > Let me ask you: how do you prepare your website for the complete collapse of western society? How did we go from 'you could lose your AWS account' to 'complete collapse of western society'? Do websites even matter in that context? > Second, preparing for the disappearance of AWS is even more silly. What's silly is not thinking ahead.
- psychoslave 1y ago
  >Let me ask you: how do you prepare your website for the complete collapse of western society? That's the main topic that going through my mind lately, if you replace "my website" with "Wikimedia movement". We need a far better social, juridical and technical architecture regarding resilience as hostil agendas are in the rise at all level agaisnt sourced trackable global volunteer community knowledge bases.
csomar 1y ago
> Now imagine for a bit that it will never come back up. Given the current geopolitical circumstances, that's not a far fetched scenario. Especially for us-east-1; or anything in the D.C. metro area.
pmontra 1y ago
In the case of a costumer of mine the AWS outage manifested itself as Twilio failing to deliver SMSes. The fallback plan has been disabling the rotation of our two SMS providers and sending all messages with the remaing one. But what if the other one had something on AWS too? Or maybe both of them have something else vital on Azure, or Google Cloud, which will fail next week and stop our service. Who knows? For small and medium sized companies it's not easy to perform an accurate due diligency.
freetanga 1y ago
Additionally I find that most Hyperscalers are trying to lock you in, by tailoring services which are industry standard with custom features which end up building roots and making a multi-vendor or lift and shift problematic. Need to keep eyes peeled at all levels or the organization as many of these enter through day-to-day…
- jacquesm 1y ago
  Yes, they're really good at that. This is just 'embrace and extend'. We all know the third.
invalidusernam3 1y ago
What if the fall-back also never comes back up?
ho_schi 1y ago
The internet is a weak infrastructure, relying on a few big cables and data centers. And through AWS and Cloudflare it has become worse? Was it ever true, that the internet is resilient? I doubt. Resilient systems work autonomously and can synchronize - but don't need to synchronize. * Git is resilient. * Native E-Mail clients - with local storage enabled - are somewhat resilient. * A local package repository is - somewhat resilient. * A local file-sharing app (not Warp/ Magic-Wormhole -> needs relay) is resilient if it uses only local WiFi or Bluetooth. We're building weak infrastructure. A lot of stuff shall work locally and only optionally use the internet.
- CaptainOfCoit 1y ago
  The internet seems resilient enough for all intents and purposes, we haven't had a global internet-wide catastrophe impacting the entire internet as far as I know, but we have gotten close to it sometimes (thanks BGP). But the web, that's the fragile, centralized and weak point currently, and seems to be what you're referring to rather. Maybe nitpicky, but I feel like it's important to distinguish between "the web" and "the internet".
  - jacquesm 1y ago
    You are absolutely correct but this distinction is getting less and less important, everything is using APIs nowadays, including lots of stuff that is utterly invisible until it goes down.
  - ho_schi 1y ago
    Sweden and the “Coop” disaster: https://www.bbc.com/news/technology-57707530 https://www.bbc.com/news/technology-57707530 That because people trust and hope blindly. They believe IT is for saving money? It isn’t. They coupled their cash registers onto an American cloud service. The couldn’t even pay in cash. It usually gets worse, when not outages happens for some time. Because that increases blind trust.
    - CaptainOfCoit 1y ago
rco8786 1y ago
If AWS goes down unexpectedly and never comes back up it's much more likely that we're in the middle of some enormous global conflict where day to day survival takes priority over making your app work than AWS just deciding to abandon their cloud business on a whim.
- apexalpha 1y ago
  Or Trump decided your country does not deserve it.
  - bombcar 1y ago
    Or Bezos.
    - dr-smooth 1y ago
      Or Bezos selling his soul to the Orange Devil and kicking you off when the Conman-in-chief puts the squeeze on some other aspect of Bezos' business empire
- CaptainOfCoit 1y ago
  Can also be much easier than that. Say you live in Mexico, hosting servers with AWS in the US because you have US customers. But suddenly the government decides to place sanctions on Mexico, and US entities are no longer allowed to do business with Mexicans, so all Mexican AWS accounts get shut down. For you as a Mexican the end results is the same, AWS went away, and considering there already is a list of countries that cannot use AWS, GitHub and a bunch of other "essential" services, it's not hard to imagine that that list might grow in the future.
- chasd00 1y ago
  what's most realistic is something like a major scandal at AWS. The FBI seizes control and no bytes come in our out until the investigation is complete. A multi-year total outage effectively.
Keyframe 1y ago
At least we've got github steady with our code and IaaC, right? Right?!
OfflineSergio 1y ago
I don't think its worth it, but lets say I did it, what if others that I depend on dont do it? I still won't be fully functional and only one of us have spent a bunch of money.
bschne 1y ago
I find this hard to judge in the abstract, but I'm not quite convinced the situation for the modal company today is worse than their answer to "what if your colo rack catches fire" would have been twenty years ago.
- jacquesm 1y ago
  > "what if your colo rack catches fire" I've actually had that. https://www.webmasterworld.com/webmaster/3663978.htm https://www.webmasterworld.com/webmaster/3663978.htm
  - bschne 1y ago
    I used to work at an SME that ran ~everything on its own colo'd hardware, and while it never got this bad, there were a couple instances of the CTO driving over to the dc because the oob access to some hung up server wasn't working anymore. Fun times...
    - pluto_modadic 1y ago
      oh hey, I've bricked a server remotely and had to drive 45 minutes to the DC to get badged in and reboot things :)
      - quesera 1y ago
        Reminiscing: this was a rite of passage for pre-cloud remote systems administrators. Proper hardware (Sun, Cisco) had a serial management interface (ideally "lights-out") which could be used to remedy many kinds of failures. Plus a terminal server with a dial-in modem on a POTS line (or adequate fakery), in case the drama took out IP routing. Then came Linux on x86, and it took waytoomanyyears for the hardware vendors to outgrow the platform's microsoft local-only deployment model. Aside from Dell and maybe Supermicro, I'm not sure if they ever worked it out. Then came the cloud. Ironically, all of our systems are up and happy today, but services that rely on partner integrations are down. The only good thing about this is that it's not me running around trying to get it fixed. :)
rglover 1y ago
It would behoove a lot of devs to learn the basics of Linux sysadmin and how to setup a basic deployment with a VPS. Once you understand that, you'll realize how much of "modern infra" is really just a mix of over-reliance on AWS and throwing compute at underperforming code. Our addiction to complexity (and burning money on the illusion of infinite stability) is already and will continue to strangle us.
kbar13 1y ago
the correct answer for those companies is "we have it on the roadmap but for right now accept the risk"
saltyoldman 1y ago
Contrast this with the top post.
JamesSwift 1y ago
What good is jumping through extraordinary hoops to be multi cloud if docker, netlify, stripe, intercom, npm, etc all go down along with us-east-1?
- fisf 1y ago
  Because you should not depend on one payment provider and pull unvendored images, packages, etc directly into your deployment. There is no reason to have such brittle infra.
  - JamesSwift 1y ago
    Sure, but at that point you go from bog standard to "enterprise grade redundancy for every single point of failure" which I can assure you is more heavily engineered than many enterprises (source: see current outage). Its just not worth the manpower and dollars for a vast majority of businesses.
    - fisf 1y ago
      Pulling unvetted stuff from docker hub, npm, etc. is not a question of redundancy.
      - JamesSwift 1y ago
        OK, you pull it to your own repo. Now where do you store it? Do you also have fallback stores for that? What about the things which arent vendorable, ie external services?
bongodongobob 1y ago
You simply cannot avoid it. There are so many applications and services that use AWS. Companies cant sit on 100% in-house software stacks.

hvb2 1y ago

> The internet got its main strengths from the fact that it was completely decentralized. Decentralized in terms of many companies making up the internet. Yes we've seen heavy consolidation in now having less than 10 companies make up the bulk of the internet. The problem here isn't caused by companies chosing one cloud provider over the other. It's the economies of scale leading us to few large companies in any sector.

jacquesm 1y ago
I think one reason is that people are just bad at statistics. Chance of materialization * impact = small. Sure. Over a short enough time that's true for any kind of risk. But companies tend to live for years, decades even and sometimes longer than that. If we're going to put all of those precious eggs in one basket, as long as the basket is substantially stronger than the eggs we're fine, right? Until the day someone drops the basket. And over a long enough time span all risks eventually materialize. So we're playing this game, and usually we come out ahead. But trust me, 10 seconds after this outage is solved everybody will have forgotten about the possibility.
- hvb2 1y ago
  Absolutely, but the cost of perfection (100% uptime in this case) is infinite. As long as the outages are rare enough and you automatically fail over to a different region, what's the problem?
  - jacquesm 1y ago
    Often simply the lack of a backup outside of the main cloud account.
    - hvb2 1y ago
      Sure, but on a typical outage how likely is it that you'll have that all up and running before the outage is resolved? And secondly, how often do you create that backup and are you willing to lose the writes since the last backup? That backup is absolutely something people should have, but I doubt those are ever used to bring a service back up. That would be a monumental failure of your hosting provider (colo/cloud/whatever)
      - jacquesm 1y ago
        > Sure, but on a typical outage how likely is it that you'll have that all up and running before the outage is resolved? Not, but if some Amazon flunky decides to kill your account to protect the Amazon brand then you will at least survive, even if you'll lose some data.
lentil_soup 1y ago
> Decentralized in terms of many companies making up the internet Not companies, the protocols are decentralized and at some point it was mostly non-companies. Anyone can hook up a computer and start serving requests which was/is a radical concept, we've lost a lot, unfortunately
- hvb2 1y ago
  No we've not lost that at all. Nobody prevents you from doing that. We have put more and more services on fewer and fewer vendors. But that's the consolidation and cost point.
  - IlikeKitties 1y ago
    > No we've not lost that at all. Nobody prevents you from doing that. May I introduce you to our Lord and Slavemaster CGNAT?
    - yupyupyups 1y ago
      That depends on who your ISP is.
    - otterley 1y ago
      There’s more than one way to get a server on the Internet. You can pay a local data center to put your machine in one of their racks.
    - tcfhgj 1y ago
      may I introduce you to ipv6?
      - IlikeKitties 1y ago
        Try hosting an E-Mail Server behind CGNat and send an E-Mail to...anyone really.
        1y ago
psychoslave 1y ago
Well, that is exactly what resilient distributed network are about. Not that much the technical details we implement them through, but the social relationship and balanced in political decision power. Be it a company or a state, concentration of power that exeeds the needs for their purpose to function by a large margin is always a sure way to spread corruption, create feedback loop in single point of failure, and is buying everyone a ticket to some dystopian reality with a level of certainty that beats anything that a SLA will ever give us.
chasd00 1y ago
Decentralized with respect to connectivity. If a construction crew cuts a fiber bundle routing protocols will route around the damage and packets keep showing up at the destination. Or, only a localized group of users will be affected. That level of decentralization is not what we have at higher levels in the stack with AWS being a good example. Even connectivity has it's points of failure. I've touched with my own hands fiber runs that, with a few quick snips from a wire cutter, could bring sizable portions of the Internet offline. Granted that was a long time ago so those points of failure may no longer exist.

padjo 1y ago

Planning for an AWS outage is a complete waste of time and energy for most companies. Yes it does happen but very rarely to the tune of a few hours every 5-10 years. I can almost guarantee that whatever plans you have won’t get you fully operational faster than just waiting for AWS to fix it.

jacquesm 1y ago
Thank you for illustrating my point. You didn't even bother to read the second paragraph.
- shawabawa3 1y ago
  > Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically eroding that strength. my business contingency plan for "AWS shuts down and never comes back up" is to go bankrupt
  - jacquesm 1y ago
    Is that also your contingency plan for 'user uploads objectionable content and alerts Amazon to get your account shut down'? Make sure you let your investors know.
    - padjo 1y ago
      If your mitigation for that risk is to have an elaborate plan to move to a different cloud provider, where the same problem can just happen again, then you’re doing an awful job of risk management.
      - jacquesm 1y ago
        > If your mitigation for that risk is to have an elaborate plan to move to a different cloud provider, where the same problem can just happen again, then you’re doing an awful job of risk management. Where did I say that? If I didn't say it: could you please argue in good faith. Thank you.

raincole 1y ago

Most companies just aren't important enough to worry about "AWS never come back up." Planning for this case is just like planning for a terrorist blowing up your entire office. If you're the Pentagon sure you'd better have a plan for that. But most companies are not the Pentagon.

Frieren 1y ago
> Most companies just aren't important enough to worry about "AWS never come back up." But a large enough of "not too big to fail" companies become a too big to fail event. Too many medium sized companies have a total dependency on AWS or Google or Microsoft, if not several at the same time. We live in a increasingly fragile society, one step closer to critical failure because big tech is not regulated in the same way than other infrastructure.
- raincole 1y ago
  Well I agree. I kinda think the AI apocalypse would not be like a skynet killing us, but a malware be patched onto all the Tesla and causes one million crashes tomorrow morning.
  - swader999 1y ago
    Battery fires.
- coffeebeqn 1y ago
  Many have a hard dependency on AWS && Google && Microsoft!
paulddraper 1y ago
Exactly. And FWIW, "AWS is down"....only one region (out of 36) of AWS is down. You can do the multi-region failover, though that's still possibly overkill for most.

anal_reactor 1y ago

First, planning for AWS outage is pointless. Unless you provide service of national security or something, your customers are going to understand that when there's global internet outage your service doesn't work either. The cost of maintaining a working failover across multiple cloud providers is just too high compared to potential benefits. It's astonishing that so few eningeers understand the fact that maintaining a technically beautiful solution costs time and money, which might not make a justified business case. Second, preparing for the disappearance of AWS is even more silly. The chance that it will happen are orders of magnitude smaller than the chance that the cost of preparing for such an event will kill your business. Let me ask you: how do you prepare your website for the complete collapse of western society? Will you be able to adapt your business model to post-apocalyptic world where it's only cockroaches?

jacquesm 1y ago
> Let me ask you: how do you prepare your website for the complete collapse of western society? How did we go from 'you could lose your AWS account' to 'complete collapse of western society'? Do websites even matter in that context? > Second, preparing for the disappearance of AWS is even more silly. What's silly is not thinking ahead.
psychoslave 1y ago
>Let me ask you: how do you prepare your website for the complete collapse of western society? That's the main topic that going through my mind lately, if you replace "my website" with "Wikimedia movement". We need a far better social, juridical and technical architecture regarding resilience as hostil agendas are in the rise at all level agaisnt sourced trackable global volunteer community knowledge bases.

csomar 1y ago

> Now imagine for a bit that it will never come back up. Given the current geopolitical circumstances, that's not a far fetched scenario. Especially for us-east-1; or anything in the D.C. metro area.

pmontra 1y ago

In the case of a costumer of mine the AWS outage manifested itself as Twilio failing to deliver SMSes. The fallback plan has been disabling the rotation of our two SMS providers and sending all messages with the remaing one. But what if the other one had something on AWS too? Or maybe both of them have something else vital on Azure, or Google Cloud, which will fail next week and stop our service. Who knows? For small and medium sized companies it's not easy to perform an accurate due diligency.

freetanga 1y ago

Additionally I find that most Hyperscalers are trying to lock you in, by tailoring services which are industry standard with custom features which end up building roots and making a multi-vendor or lift and shift problematic. Need to keep eyes peeled at all levels or the organization as many of these enter through day-to-day…

jacquesm 1y ago
Yes, they're really good at that. This is just 'embrace and extend'. We all know the third.

invalidusernam3 1y ago

What if the fall-back also never comes back up?

ho_schi 1y ago

The internet is a weak infrastructure, relying on a few big cables and data centers. And through AWS and Cloudflare it has become worse? Was it ever true, that the internet is resilient? I doubt. Resilient systems work autonomously and can synchronize - but don't need to synchronize. * Git is resilient. * Native E-Mail clients - with local storage enabled - are somewhat resilient. * A local package repository is - somewhat resilient. * A local file-sharing app (not Warp/ Magic-Wormhole -> needs relay) is resilient if it uses only local WiFi or Bluetooth. We're building weak infrastructure. A lot of stuff shall work locally and only optionally use the internet.

CaptainOfCoit 1y ago
The internet seems resilient enough for all intents and purposes, we haven't had a global internet-wide catastrophe impacting the entire internet as far as I know, but we have gotten close to it sometimes (thanks BGP). But the web, that's the fragile, centralized and weak point currently, and seems to be what you're referring to rather. Maybe nitpicky, but I feel like it's important to distinguish between "the web" and "the internet".
- jacquesm 1y ago
  You are absolutely correct but this distinction is getting less and less important, everything is using APIs nowadays, including lots of stuff that is utterly invisible until it goes down.
- ho_schi 1y ago
  Sweden and the “Coop” disaster: https://www.bbc.com/news/technology-57707530 https://www.bbc.com/news/technology-57707530 That because people trust and hope blindly. They believe IT is for saving money? It isn’t. They coupled their cash registers onto an American cloud service. The couldn’t even pay in cash. It usually gets worse, when not outages happens for some time. Because that increases blind trust.
  - CaptainOfCoit 1y ago

rco8786 1y ago

If AWS goes down unexpectedly and never comes back up it's much more likely that we're in the middle of some enormous global conflict where day to day survival takes priority over making your app work than AWS just deciding to abandon their cloud business on a whim.

apexalpha 1y ago
Or Trump decided your country does not deserve it.
- bombcar 1y ago
  Or Bezos.
  - dr-smooth 1y ago
    Or Bezos selling his soul to the Orange Devil and kicking you off when the Conman-in-chief puts the squeeze on some other aspect of Bezos' business empire
CaptainOfCoit 1y ago
Can also be much easier than that. Say you live in Mexico, hosting servers with AWS in the US because you have US customers. But suddenly the government decides to place sanctions on Mexico, and US entities are no longer allowed to do business with Mexicans, so all Mexican AWS accounts get shut down. For you as a Mexican the end results is the same, AWS went away, and considering there already is a list of countries that cannot use AWS, GitHub and a bunch of other "essential" services, it's not hard to imagine that that list might grow in the future.
chasd00 1y ago
what's most realistic is something like a major scandal at AWS. The FBI seizes control and no bytes come in our out until the investigation is complete. A multi-year total outage effectively.

Keyframe 1y ago

At least we've got github steady with our code and IaaC, right? Right?!

OfflineSergio 1y ago

I don't think its worth it, but lets say I did it, what if others that I depend on dont do it? I still won't be fully functional and only one of us have spent a bunch of money.

bschne 1y ago

I find this hard to judge in the abstract, but I'm not quite convinced the situation for the modal company today is worse than their answer to "what if your colo rack catches fire" would have been twenty years ago.

jacquesm 1y ago
> "what if your colo rack catches fire" I've actually had that. https://www.webmasterworld.com/webmaster/3663978.htm https://www.webmasterworld.com/webmaster/3663978.htm
- bschne 1y ago
  I used to work at an SME that ran ~everything on its own colo'd hardware, and while it never got this bad, there were a couple instances of the CTO driving over to the dc because the oob access to some hung up server wasn't working anymore. Fun times...
  - pluto_modadic 1y ago
    oh hey, I've bricked a server remotely and had to drive 45 minutes to the DC to get badged in and reboot things :)
    - quesera 1y ago
      Reminiscing: this was a rite of passage for pre-cloud remote systems administrators. Proper hardware (Sun, Cisco) had a serial management interface (ideally "lights-out") which could be used to remedy many kinds of failures. Plus a terminal server with a dial-in modem on a POTS line (or adequate fakery), in case the drama took out IP routing. Then came Linux on x86, and it took waytoomanyyears for the hardware vendors to outgrow the platform's microsoft local-only deployment model. Aside from Dell and maybe Supermicro, I'm not sure if they ever worked it out. Then came the cloud. Ironically, all of our systems are up and happy today, but services that rely on partner integrations are down. The only good thing about this is that it's not me running around trying to get it fixed. :)

rglover 1y ago

It would behoove a lot of devs to learn the basics of Linux sysadmin and how to setup a basic deployment with a VPS. Once you understand that, you'll realize how much of "modern infra" is really just a mix of over-reliance on AWS and throwing compute at underperforming code. Our addiction to complexity (and burning money on the illusion of infinite stability) is already and will continue to strangle us.

kbar13 1y ago

the correct answer for those companies is "we have it on the roadmap but for right now accept the risk"

saltyoldman 1y ago

Contrast this with the top post.

JamesSwift 1y ago

What good is jumping through extraordinary hoops to be multi cloud if docker, netlify, stripe, intercom, npm, etc all go down along with us-east-1?

fisf 1y ago
Because you should not depend on one payment provider and pull unvendored images, packages, etc directly into your deployment. There is no reason to have such brittle infra.
- JamesSwift 1y ago
  Sure, but at that point you go from bog standard to "enterprise grade redundancy for every single point of failure" which I can assure you is more heavily engineered than many enterprises (source: see current outage). Its just not worth the manpower and dollars for a vast majority of businesses.
  - fisf 1y ago
    Pulling unvetted stuff from docker hub, npm, etc. is not a question of redundancy.
    - JamesSwift 1y ago
      OK, you pull it to your own repo. Now where do you store it? Do you also have fallback stores for that? What about the things which arent vendorable, ie external services?

bongodongobob 1y ago

You simply cannot avoid it. There are so many applications and services that use AWS. Companies cant sit on 100% in-house software stacks.