5 ms·

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting

I think now is a good time to reiterate the danger of companies just throwing all of their operational resilience and sustainability over the wall and trusting someone else with their entire existence. It's wild to me that so many high performing businesses simply don't have a plan for when the cloud goes down. Some of my contacts are telling me that these outages have teams of thousands of people completely prevented from working and tens of million dollars of profit are simply vanishing since the start of the outage this morning. And now institutions like government and banks are throwing their entire capability into the cloud with no recourse or recovery plan. It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down. We are incredibly blind to just trust just 3 cloud providers with the operational success of basically everything we do. Why hasn't the industry come up with an alternative?

bowmessage 5y ago
Because the majority of consumers don't know better / don't care and still buy products from companies with no backup plan. Because, really, how can any of us know better until we're burned many times over?
tw04 5y ago
>It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down. Most financial institutions are implementing their own clouds, I can't think of any major one that is reliant on public cloud to the extent transactions would stop. >Why hasn't the industry come up with an alternative? You mean like building datacenters and hosting your own gear?
- baoyu 5y ago
  > Most financial institutions are implementing their own clouds https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement
  - filmgirlcw 5y ago
    That doesn't mean what you think it means. The agreement is more of a hybrid cloud arrangement with AWS Outposts. FTA: >Core to Nasdaq’s move to AWS will be AWS Outposts, which extend AWS infrastructure, services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility. Nasdaq plans to incorporate AWS Outposts directly into its core network to deliver ultra-low-latency edge compute capabilities from its primary data center in Carteret, NJ. They are also starting small, with Nasdaq MRX This is much less about moving NASDAQ (or other exchanges) to be fully owned/maintained by Amazon, and more about wanting to take advantage of development tooling and resources and services AWS provides, but within the confines of an owned/maintained data center. I'm sure as this partnership grows, racks and racks will be in Amazon's data centers too, but this is a hybrid approach. I would also bet a significant amount of money that when NASDAQ does go full "cloud" (or hybrid, as it were), it won't be in the same US-east region co-mingling with the rest of the consumer web, but with its own redundant services and connections and networking stack. NASDAQ wants to modernize its infrastructure but it absolutely doesn't want to offload it to a cloud provider. That's why it's a hybrid partnership.
- jen20 5y ago
  Indeed I can think of several outages in the past decade in the UK of banks' own infrastructure which have led to transactions stopping for days at a time, with the predictable outcomes.
deleted 5y ago
[deleted]
jessebarton 5y ago
In my opinion there is a lack of talent in these industries for building out there own resilient systems. IT people and engineers get lazy.
- BarryMilo 5y ago
  No lazier than anyone else, there's just not enough of us, in general and per company.
- rodgerd 5y ago
  > IT people and engineers get lazy. Companies do not change their whole strategy from a capex-driven traditional self-hosting environment to opex-driven cloud hosting because their IT people are lazy; it is typically an exec-level decision.
- grumple 5y ago
  We're too busy in endless sprints to focus on things outside of our core business that don't make salespeople and executives excited.
adflux 5y ago
Because your own datacenters cant go down?
uranium 5y ago
Because the expected value of using AWS is greater than the expected value of self-hosting. It's not that nobody's ever heard of running on their own metal. Look back at what everyone did before AWS, and how fast they ran screaming away from it as soon as they could. Once you didn't have to do that any more, it's just so much better that the rare outages are worth it for the vast majority of startups. Medical devices, banks, the military, etc. should generally run on their own hardware. The next photo-sharing app? It's just not worth it until they hit tremendous scale.
- chucknthem 5y ago
  Agree with your first point. On the second though, at some point, infrastructure like AWS are going to be more reliable than what many banks, medical device operators etc can provide themselves. asking them to stay on their own hardware is asking for that industry to remain slow, bespoke and expensive.
  - uranium 5y ago
    Agreed, and it'll be a gradual switch rather than a single point, smeared across industries. Likely some operations won't ever go over, but it'll be a while before we know.
  - hn_throwaway_99 5y ago
    Hard agree with the second paragraph. It is incredibly difficult for non-tech companies to hire quality software and infrastructure engineers - they usually pay less and the problems aren't as interesting.
smugglerFlynn 5y ago
Many of those businesses wouldn’t have existed in the first place without simplicity offered by cloud.
stfp 5y ago
> tens of million dollars of profit are simply vanishing vanishing or delayed six hours? I mean
- naikrovek 5y ago
  money people think of money very weirdly. when they predict they will get more than they actually get, they call it "loss" for some reason, and when they predict they will get less than they actually get, it's called ... well I don't know what that's called but everyone gets bonuses.
- lp0_on_fire 5y ago
  6 hours of downtime often means 6 hours of paying employees to stand around which adds up rather quickly.
nprz 5y ago
So you're saying companies should start moving their infrastructure to the blockchain?
- tornato7 5y ago
  Ethereum has gone 5 years without a single minute of downtime, so if it's extreme reliability you're going for I don't think it can be beaten.
  - amusedcyclist 5y ago
    Easy to be reliable when nobody uses it
rbetts 5y ago
We're too busy working generating our own electricity and designing our own CPUs.
300bps 5y ago
This appears to be a single region outage - us-east-1. AWS supports as much redundancy as you want. You can be redundant between multiple Availability Zones in a single Region or you can be redundant among 1, 2 or even 25 regions throughout the world. Multiple-region redundancy costs more both in initial planning/setup as well as monthly fees so a lot of AWS customers choose to just not do it.
george3d6 5y ago
This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the problem is hard. AWS has fewer outages in one zone alone than the best self-hosted institutions, your facebooks and petagons. In-house servers would lead to an insane amount of outage. And guess what? AWS (and all other IAAS providers) will beg you to use multiple region because of this. The team/person that has millions of dollars a day staked on a single AWS region is an idiot and could not be entrusted to order a gaming PC from newegg, let alone run an in-house datacenter. edit: I will add that AWS specifically is meh and I wouldn't use it myself, there's better IASS. But it's insanity to even imagine self-hosted is more reliable than using even the shittiest of IASS providers.
- roody15 5y ago
  Quick follow up. I once used a IASS provider (hyperstreet) that was terrible. Long story short provider ended closing shop and the owner of the company now sells real estate in California. Was a nightmare recovering data. Even when the service was operational was sub par. Just saying perhaps the “shittiest” providers may not be more reliable.
- deleted 5y ago
  [deleted]
- jgwil2 5y ago
  > In-house servers would lead to an insane amount of outage. That might be true, but the effects of any given outage would be felt much less widely. If Disney has an outage, I can just find a movie on Netflix to watch instead. But now if one provider goes down, it can take down everything. To me, the problem isn't the cloud per se, it's one player's dominance in the space. We've taken the inherently distributed structure of the internet and re-centralized it, losing some robustness along the way.
  - 5y ago
tcgv 5y ago
> Why hasn't the industry come up with an alternative? The cloud is the solution to self managed data centers. Their value proposition is appealing: Focus on your core business and let us handle infrastructure for you. This fits the needs of most small and medium sized businesses, there's no reason not to use the cloud and spend time and money on building and operating private data centers when the (perceived) chances of outages are so small. Then, companies grow to a certain size where the benefits of having a self managed data center begins to outweight not having one. But at this point this becomes more of a strategic/political decision than merely a technical one, so it's not an easy shift.
- noogle 5y ago
  Why is the comparison always between public cloud and building your own data center from scratch? There is a middle ground of bare-metal hosting. Abstract away the hardware and networking. Do the rest yourself.
p1necone 5y ago
Do you think they'd manage their own infra better? Are you suggesting they pay for a fully redundant second implementation on another provider? How much extra cost would that be vs eating an outage very infrequently?
seeEllArr 5y ago
They have, it's called hosting on-premesis, and it's even less reliable than cloud providers.
olingern 5y ago
People are so quick to forget how things were before behemoths like AWS, Google Cloud, and Azure. Not all things are free and the outage the internet is experiencing is the risk users signed up for. If you would like to go back to the days of managing your own machines, be my guest. Remember those machines also live somewhere and were/are subject to the same BGP and routing issues we've seen over the past couple of years. Personally, I'll deal with outages a few times a year for the peace of mind that there's a group of really talented people looking into for me.
xwdv 5y ago
There is an alternative: A true network cloud. This is what Cloudflare will eventually become.
jacobsenscott 5y ago
We have or had alternatives - rackspace, linode, digital ocean, in the past there were many others, self hosting is still an option. But the big three just do it better. The alternatives are doomed to fail. If you use anything other than the big three you risk not just more outages, but your whole provider going out of business overnight. If the companies at the scale you are talking about do not have multi-region and multi service (aws to azure for example) failover that's their fault, and nobody else's.
tuldia 5y ago
> Why hasn't the industry come up with an alternative? We used to have that, some companies still have the capability and know-how to build and run infrastructure that is reliable, distributed across many hosting providers before "cloud" became the "norm", but it goes along with "use or lose it".
commandlinefan 5y ago
Well, if you're web-based, there's never really been any better alternative. Even before "the cloud", you had to be hosted in a datacenter somewhere if you wanted enough bandwidth to service customers, as well as have somebody who would make sure the power stayed on 24/7. The difference now is that there used to be thousands of ISP's so one outage wouldn't get as much news coverage, but it would also probably last longer because you wouldn't have a team of people who know what to look for like Amazon (probably?) does.

bowmessage 5y ago

Because the majority of consumers don't know better / don't care and still buy products from companies with no backup plan. Because, really, how can any of us know better until we're burned many times over?

tw04 5y ago

>It seems bad now but I wonder how much worse it might be when no one actually has access to money because all financial traffic is going through AWS and it goes down. Most financial institutions are implementing their own clouds, I can't think of any major one that is reliant on public cloud to the extent transactions would stop. >Why hasn't the industry come up with an alternative? You mean like building datacenters and hosting your own gear?

baoyu 5y ago
> Most financial institutions are implementing their own clouds https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement https://www.nasdaq.com/Nasdaq-AWS-cloud-announcement
- filmgirlcw 5y ago
  That doesn't mean what you think it means. The agreement is more of a hybrid cloud arrangement with AWS Outposts. FTA: >Core to Nasdaq’s move to AWS will be AWS Outposts, which extend AWS infrastructure, services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility. Nasdaq plans to incorporate AWS Outposts directly into its core network to deliver ultra-low-latency edge compute capabilities from its primary data center in Carteret, NJ. They are also starting small, with Nasdaq MRX This is much less about moving NASDAQ (or other exchanges) to be fully owned/maintained by Amazon, and more about wanting to take advantage of development tooling and resources and services AWS provides, but within the confines of an owned/maintained data center. I'm sure as this partnership grows, racks and racks will be in Amazon's data centers too, but this is a hybrid approach. I would also bet a significant amount of money that when NASDAQ does go full "cloud" (or hybrid, as it were), it won't be in the same US-east region co-mingling with the rest of the consumer web, but with its own redundant services and connections and networking stack. NASDAQ wants to modernize its infrastructure but it absolutely doesn't want to offload it to a cloud provider. That's why it's a hybrid partnership.
jen20 5y ago
Indeed I can think of several outages in the past decade in the UK of banks' own infrastructure which have led to transactions stopping for days at a time, with the predictable outcomes.

deleted 5y ago

[deleted]

jessebarton 5y ago

In my opinion there is a lack of talent in these industries for building out there own resilient systems. IT people and engineers get lazy.

BarryMilo 5y ago
No lazier than anyone else, there's just not enough of us, in general and per company.
rodgerd 5y ago
> IT people and engineers get lazy. Companies do not change their whole strategy from a capex-driven traditional self-hosting environment to opex-driven cloud hosting because their IT people are lazy; it is typically an exec-level decision.
grumple 5y ago
We're too busy in endless sprints to focus on things outside of our core business that don't make salespeople and executives excited.

adflux 5y ago

Because your own datacenters cant go down?

uranium 5y ago

Because the expected value of using AWS is greater than the expected value of self-hosting. It's not that nobody's ever heard of running on their own metal. Look back at what everyone did before AWS, and how fast they ran screaming away from it as soon as they could. Once you didn't have to do that any more, it's just so much better that the rare outages are worth it for the vast majority of startups. Medical devices, banks, the military, etc. should generally run on their own hardware. The next photo-sharing app? It's just not worth it until they hit tremendous scale.

chucknthem 5y ago
Agree with your first point. On the second though, at some point, infrastructure like AWS are going to be more reliable than what many banks, medical device operators etc can provide themselves. asking them to stay on their own hardware is asking for that industry to remain slow, bespoke and expensive.
- uranium 5y ago
  Agreed, and it'll be a gradual switch rather than a single point, smeared across industries. Likely some operations won't ever go over, but it'll be a while before we know.
- hn_throwaway_99 5y ago
  Hard agree with the second paragraph. It is incredibly difficult for non-tech companies to hire quality software and infrastructure engineers - they usually pay less and the problems aren't as interesting.

smugglerFlynn 5y ago

Many of those businesses wouldn’t have existed in the first place without simplicity offered by cloud.

stfp 5y ago

> tens of million dollars of profit are simply vanishing vanishing or delayed six hours? I mean

naikrovek 5y ago
money people think of money very weirdly. when they predict they will get more than they actually get, they call it "loss" for some reason, and when they predict they will get less than they actually get, it's called ... well I don't know what that's called but everyone gets bonuses.
lp0_on_fire 5y ago
6 hours of downtime often means 6 hours of paying employees to stand around which adds up rather quickly.

nprz 5y ago

So you're saying companies should start moving their infrastructure to the blockchain?

tornato7 5y ago
Ethereum has gone 5 years without a single minute of downtime, so if it's extreme reliability you're going for I don't think it can be beaten.
- amusedcyclist 5y ago
  Easy to be reliable when nobody uses it

rbetts 5y ago

We're too busy working generating our own electricity and designing our own CPUs.

300bps 5y ago

This appears to be a single region outage - us-east-1. AWS supports as much redundancy as you want. You can be redundant between multiple Availability Zones in a single Region or you can be redundant among 1, 2 or even 25 regions throughout the world. Multiple-region redundancy costs more both in initial planning/setup as well as monthly fees so a lot of AWS customers choose to just not do it.

george3d6 5y ago

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the problem is hard. AWS has fewer outages in one zone alone than the best self-hosted institutions, your facebooks and petagons. In-house servers would lead to an insane amount of outage. And guess what? AWS (and all other IAAS providers) will beg you to use multiple region because of this. The team/person that has millions of dollars a day staked on a single AWS region is an idiot and could not be entrusted to order a gaming PC from newegg, let alone run an in-house datacenter. edit: I will add that AWS specifically is meh and I wouldn't use it myself, there's better IASS. But it's insanity to even imagine self-hosted is more reliable than using even the shittiest of IASS providers.

roody15 5y ago
Quick follow up. I once used a IASS provider (hyperstreet) that was terrible. Long story short provider ended closing shop and the owner of the company now sells real estate in California. Was a nightmare recovering data. Even when the service was operational was sub par. Just saying perhaps the “shittiest” providers may not be more reliable.
deleted 5y ago
[deleted]
jgwil2 5y ago
> In-house servers would lead to an insane amount of outage. That might be true, but the effects of any given outage would be felt much less widely. If Disney has an outage, I can just find a movie on Netflix to watch instead. But now if one provider goes down, it can take down everything. To me, the problem isn't the cloud per se, it's one player's dominance in the space. We've taken the inherently distributed structure of the internet and re-centralized it, losing some robustness along the way.
- 5y ago

tcgv 5y ago

> Why hasn't the industry come up with an alternative? The cloud is the solution to self managed data centers. Their value proposition is appealing: Focus on your core business and let us handle infrastructure for you. This fits the needs of most small and medium sized businesses, there's no reason not to use the cloud and spend time and money on building and operating private data centers when the (perceived) chances of outages are so small. Then, companies grow to a certain size where the benefits of having a self managed data center begins to outweight not having one. But at this point this becomes more of a strategic/political decision than merely a technical one, so it's not an easy shift.

noogle 5y ago
Why is the comparison always between public cloud and building your own data center from scratch? There is a middle ground of bare-metal hosting. Abstract away the hardware and networking. Do the rest yourself.

p1necone 5y ago

Do you think they'd manage their own infra better? Are you suggesting they pay for a fully redundant second implementation on another provider? How much extra cost would that be vs eating an outage very infrequently?

seeEllArr 5y ago

They have, it's called hosting on-premesis, and it's even less reliable than cloud providers.

olingern 5y ago

People are so quick to forget how things were before behemoths like AWS, Google Cloud, and Azure. Not all things are free and the outage the internet is experiencing is the risk users signed up for. If you would like to go back to the days of managing your own machines, be my guest. Remember those machines also live somewhere and were/are subject to the same BGP and routing issues we've seen over the past couple of years. Personally, I'll deal with outages a few times a year for the peace of mind that there's a group of really talented people looking into for me.

xwdv 5y ago

There is an alternative: A true network cloud. This is what Cloudflare will eventually become.

jacobsenscott 5y ago

We have or had alternatives - rackspace, linode, digital ocean, in the past there were many others, self hosting is still an option. But the big three just do it better. The alternatives are doomed to fail. If you use anything other than the big three you risk not just more outages, but your whole provider going out of business overnight. If the companies at the scale you are talking about do not have multi-region and multi service (aws to azure for example) failover that's their fault, and nobody else's.

tuldia 5y ago

> Why hasn't the industry come up with an alternative? We used to have that, some companies still have the capability and know-how to build and run infrastructure that is reliable, distributed across many hosting providers before "cloud" became the "norm", but it goes along with "use or lose it".

commandlinefan 5y ago

Well, if you're web-based, there's never really been any better alternative. Even before "the cloud", you had to be hosted in a datacenter somewhere if you wanted enough bandwidth to service customers, as well as have somebody who would make sure the power stayed on 24/7. The difference now is that there used to be thousands of ISP's so one outage wouldn't get as much news coverage, but it would also probably last longer because you wouldn't have a team of people who know what to look for like Amazon (probably?) does.