8 ms·

Anyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down

Anyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down. Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?

tyingq 5y ago
I think also just finally getting to the point where they have a tech debt burden comparable to everyone else. Starting fresh has its advantages...for a while.
- papito 5y ago
  Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.
  - deleted 5y ago
    [deleted]
  - willcipriano 5y ago
    Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.
    - carlmr 5y ago
      I've never seen it. It's kind of like credit card debt. The people that have it get progressively more to pay off the previous debt until they reach bankruptcy. So I do think tech debt is accurate, in that the average human deludes themselves into thinking they will pay it off at some point.
    - tyingq 5y ago
      It seems like they could at least cordon some of it off into a corral of "legacy products". Or maybe a concept of "next gen" regions that have only a subset of products.
    - papito 5y ago
      You never pay it off. It's like "the war on crime" or "the war on drugs". Don't call it a war if there is no clear win. You can never win the war, but you MUST win the battles in order to not lose it.
    - xtracto 5y ago
      I loved a quote someone else wrote here some time ago: "Hackers are just tech debt collectors" [1]. If ever there has been a true quote, it is this. [1] https://news.ycombinator.com/item?id=29039611 https://news.ycombinator.com/item?id=29039611
    - rrrrrrrrrrrryan 5y ago
      A well-maintained car can drive for a very long time, but for a poorly maintained car, restoring it to like-new condition rarely makes financial sense. It's usually cheaper to just buy a new car, and/or do the bare-minimum maintenance until it dies completely (then buy a new car). I'm not sure tech debt can ever be fully paid down. You can pay off a bit, and you can stop the debt from accumulating further, but the only way to realistically unburden yourself of a really horrible, big pile of tech debt is to just rebuild the thing from scratch. Or don't, and just keep making money while you wait for the business to fail. (From an investor's perspective, this is the equivalent of riding the car into the ground.)
    - stjohnswarts 5y ago
      I think the term is fine. I don't don't know if is the "ever accumulating" version is correct as at some point it levels off in my experience. At some point things break and you have to take care of it.
kaidon 5y ago
I was thinking something similar. The FAANG interview has been optimized to punish candidates with actual experience in these problems in favor of those that can solve leetcode problems and regurgitate design architectures from youtube. This filters out a lot of engineers with grit that can actually work through tough real world (IT) problems.
- gonzo41 5y ago
  Those people, the gritty ones who can't make it through the interview hoops are busy integrating companies into the cloud providers. Now they're taking it easy whilst the new kids are getting a first class lesson in what grit is.
- aeyes 5y ago
  I don't even work at FAANG but even we have our own in-house grown compute platforms with layers and layers of abstractions and enormous complexity which were built over several years by hundreds of engineers. These don't resemble public cloud or open source solutions at all, ideas are similar at best. You can have a lot of real world small company knowledge but the moment you walk through the door this experience is worthless, all that counts is how fast you can adapt and how good your understanding of the basics is. Look at some of the code that was open sourced by Yahoo years ago. Other large engineering organizations today have technology which far exceeds the complexity of what was made public by Yahoo a decade ago. Unfortunately Yahoo is pretty much the only example of a large company open sourcing big portions of their platform code.
ceejayoz 5y ago
https://en.wikipedia.org/wiki/Availability_bias https://en.wikipedia.org/wiki/Availability_bias Twitter used to go down so frequently the "fail whale" was a whole cultural thing. Big AWS outages are rare, but hardly new. https://www.theregister.com/2017/03/01/aws_s3_outage/ https://www.theregister.com/2017/03/01/aws_s3_outage/ https://arstechnica.com/information-technology/2012/10/amazon-web-services-outage-once-again-shows-reality-behind-the-cloud/ https://arstechnica.com/information-technology/2012/10/amazo...
- SamuelAdams 5y ago
  Yep I remember the same issues with Reddit. They’re fairly stable now, but up until about 2015 they had outages maybe once a month or so.
mschuster91 5y ago
I'd say the scale is the problem, not the people. Every service you mentioned (maybe sans Gmail, the only additions I remember in the last twelve years are a new UI and Hangouts) has bolted on so many features over the last years: AWS was virtual machines + SDNs in the beginning as Amazon only intended to sell spare capacity on their own servers, now it's a global one-stop-shop for everything that can be done on the Internet. Facebook was a social media feed, now it's event coordination, groups, chat, image and media hosting at global scale. And apparently, no one at these organizations ever thought about re-working their infrastructure with "lessons learned over the last decade" in mind. Every new feature was simply bolted on, on top of an infrastructure that was hardly even envisioned to ever become the scale they are today. And that sort of refactoring costs serious amounts of money and developer time, not to mention that it doesn't make sense to develop new features on a code base that's going to be shut down in a year, so management doesn't approve it out of a fear they will be "out-featured" by a competitor and cannot react (=copy, like Instagram's Stories that were a clear rip-off from Snapchat) in time or that their own PKI/OKR goals and with it their bonus payments won't get hit. That mindset/scale issue is also why IBM mainframes are still so common, why travel PIRs seem to be stuck in formats over half a century old or why "put CSV files on an FTP server" is the standard on bank transfers... big corporate/government clients pay a shitload of money for virtualized mainframes on new hardware that still can run the 70s-era code and even more money for people able to speak COBOL, because that is still cheaper than the alternative - reworking everything from scratch on a modern foundation, testing data integrity and edge cases, revise interfaces to hundreds or thousands of clients. Hell, even Internet standards have the same problem... we are still using protocols like BGP that have been around since before I was born, and tacked on security only a few years ago after a couple of fat-finger incidents. Modernization in such entities only tends to happen when laws or regulatory frameworks change, and then it can become a real shitshow for those at the lowest rungs of the IT ladder that have to implement them - simply take same-sex marriages and try to shoehorn them into a database that was labeled for "husband and wife", or trans/inter people with gender data represented by a boolean field.
jboggan 5y ago
As the layers of the stack increase in depth the probability that any one piece breaks the system approaches one. There will be a Singularity but it will be a horizontal asymptote instead of a vertical one.
armchairhacker 5y ago
maybe we actually have a case for decentralized “web3” networks. At least ones which really need reliability, but can’t can use whatever e.g. airplanes use (telehealth? online exams?) Of course you need to make sure the network is actually decentralized (see Solana). Or maybe you can just rely on Amazon and GCP and Azure, because surely they won’t all fail at the same time.
- colechristensen 5y ago
  The whole of Amazon isn’t falling, just some services in one region, it is easy enough to build region redundant services but most people don’t because the few hours of downtime per year don’t seem worth it. You usually just don’t need 100% uptime. “Sorry were closed, come back later” is fine.
stjohnswarts 5y ago
I wonder if we're hearing the full story. Could they be being attacked by foreign hacker farms in Russia and China just to make the US infrastructure look bad?

tyingq 5y ago

I think also just finally getting to the point where they have a tech debt burden comparable to everyone else. Starting fresh has its advantages...for a while.

papito 5y ago
Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.
- deleted 5y ago
  [deleted]
- willcipriano 5y ago
  Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.
  - carlmr 5y ago
    I've never seen it. It's kind of like credit card debt. The people that have it get progressively more to pay off the previous debt until they reach bankruptcy. So I do think tech debt is accurate, in that the average human deludes themselves into thinking they will pay it off at some point.
  - tyingq 5y ago
    It seems like they could at least cordon some of it off into a corral of "legacy products". Or maybe a concept of "next gen" regions that have only a subset of products.
  - papito 5y ago
    You never pay it off. It's like "the war on crime" or "the war on drugs". Don't call it a war if there is no clear win. You can never win the war, but you MUST win the battles in order to not lose it.
  - xtracto 5y ago
    I loved a quote someone else wrote here some time ago: "Hackers are just tech debt collectors" [1]. If ever there has been a true quote, it is this. [1] https://news.ycombinator.com/item?id=29039611 https://news.ycombinator.com/item?id=29039611
  - rrrrrrrrrrrryan 5y ago
    A well-maintained car can drive for a very long time, but for a poorly maintained car, restoring it to like-new condition rarely makes financial sense. It's usually cheaper to just buy a new car, and/or do the bare-minimum maintenance until it dies completely (then buy a new car). I'm not sure tech debt can ever be fully paid down. You can pay off a bit, and you can stop the debt from accumulating further, but the only way to realistically unburden yourself of a really horrible, big pile of tech debt is to just rebuild the thing from scratch. Or don't, and just keep making money while you wait for the business to fail. (From an investor's perspective, this is the equivalent of riding the car into the ground.)
  - stjohnswarts 5y ago
    I think the term is fine. I don't don't know if is the "ever accumulating" version is correct as at some point it levels off in my experience. At some point things break and you have to take care of it.

kaidon 5y ago

I was thinking something similar. The FAANG interview has been optimized to punish candidates with actual experience in these problems in favor of those that can solve leetcode problems and regurgitate design architectures from youtube. This filters out a lot of engineers with grit that can actually work through tough real world (IT) problems.

gonzo41 5y ago
Those people, the gritty ones who can't make it through the interview hoops are busy integrating companies into the cloud providers. Now they're taking it easy whilst the new kids are getting a first class lesson in what grit is.
aeyes 5y ago
I don't even work at FAANG but even we have our own in-house grown compute platforms with layers and layers of abstractions and enormous complexity which were built over several years by hundreds of engineers. These don't resemble public cloud or open source solutions at all, ideas are similar at best. You can have a lot of real world small company knowledge but the moment you walk through the door this experience is worthless, all that counts is how fast you can adapt and how good your understanding of the basics is. Look at some of the code that was open sourced by Yahoo years ago. Other large engineering organizations today have technology which far exceeds the complexity of what was made public by Yahoo a decade ago. Unfortunately Yahoo is pretty much the only example of a large company open sourcing big portions of their platform code.

ceejayoz 5y ago

https://en.wikipedia.org/wiki/Availability_bias https://en.wikipedia.org/wiki/Availability_bias Twitter used to go down so frequently the "fail whale" was a whole cultural thing. Big AWS outages are rare, but hardly new. https://www.theregister.com/2017/03/01/aws_s3_outage/ https://www.theregister.com/2017/03/01/aws_s3_outage/ https://arstechnica.com/information-technology/2012/10/amazon-web-services-outage-once-again-shows-reality-behind-the-cloud/ https://arstechnica.com/information-technology/2012/10/amazo...

SamuelAdams 5y ago
Yep I remember the same issues with Reddit. They’re fairly stable now, but up until about 2015 they had outages maybe once a month or so.

mschuster91 5y ago

I'd say the scale is the problem, not the people. Every service you mentioned (maybe sans Gmail, the only additions I remember in the last twelve years are a new UI and Hangouts) has bolted on so many features over the last years: AWS was virtual machines + SDNs in the beginning as Amazon only intended to sell spare capacity on their own servers, now it's a global one-stop-shop for everything that can be done on the Internet. Facebook was a social media feed, now it's event coordination, groups, chat, image and media hosting at global scale. And apparently, no one at these organizations ever thought about re-working their infrastructure with "lessons learned over the last decade" in mind. Every new feature was simply bolted on, on top of an infrastructure that was hardly even envisioned to ever become the scale they are today. And that sort of refactoring costs serious amounts of money and developer time, not to mention that it doesn't make sense to develop new features on a code base that's going to be shut down in a year, so management doesn't approve it out of a fear they will be "out-featured" by a competitor and cannot react (=copy, like Instagram's Stories that were a clear rip-off from Snapchat) in time or that their own PKI/OKR goals and with it their bonus payments won't get hit. That mindset/scale issue is also why IBM mainframes are still so common, why travel PIRs seem to be stuck in formats over half a century old or why "put CSV files on an FTP server" is the standard on bank transfers... big corporate/government clients pay a shitload of money for virtualized mainframes on new hardware that still can run the 70s-era code and even more money for people able to speak COBOL, because that is still cheaper than the alternative - reworking everything from scratch on a modern foundation, testing data integrity and edge cases, revise interfaces to hundreds or thousands of clients. Hell, even Internet standards have the same problem... we are still using protocols like BGP that have been around since before I was born, and tacked on security only a few years ago after a couple of fat-finger incidents. Modernization in such entities only tends to happen when laws or regulatory frameworks change, and then it can become a real shitshow for those at the lowest rungs of the IT ladder that have to implement them - simply take same-sex marriages and try to shoehorn them into a database that was labeled for "husband and wife", or trans/inter people with gender data represented by a boolean field.

jboggan 5y ago

As the layers of the stack increase in depth the probability that any one piece breaks the system approaches one. There will be a Singularity but it will be a horizontal asymptote instead of a vertical one.

armchairhacker 5y ago

maybe we actually have a case for decentralized “web3” networks. At least ones which really need reliability, but can’t can use whatever e.g. airplanes use (telehealth? online exams?) Of course you need to make sure the network is actually decentralized (see Solana). Or maybe you can just rely on Amazon and GCP and Azure, because surely they won’t all fail at the same time.

colechristensen 5y ago
The whole of Amazon isn’t falling, just some services in one region, it is easy enough to build region redundant services but most people don’t because the few hours of downtime per year don’t seem worth it. You usually just don’t need 100% uptime. “Sorry were closed, come back later” is fine.

stjohnswarts 5y ago

I wonder if we're hearing the full story. Could they be being attacked by foreign hacker farms in Russia and China just to make the US infrastructure look bad?