15 ms·

Is Amazon's cloud service too big to fail?

182 points by azureel 9y ago

dalbasal 9y ago
This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.
- pferde 9y ago
  > Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. I'd say that most industries do not have anyone responsible for worrying about it high enough in the management chain.
  - deleted 9y ago
    [deleted]
- tyfon 9y ago
  I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist. That should be the priority.
  - 65827 9y ago
    Why is every arrangement of characters now a "meme". That word has moved beyond devoid of meaning, at this point it's like a black hole of nothingness of a word.
    - TrickyRick 9y ago
      > A meme (/ˈmiːm/ MEEM) is an idea, behavior, or style that spreads from person to person within a culture. Seems like it's fitting here, does it not? Certain banks being "too big to fail" is an idea passed among persons within our culture.
      - sharemywin 9y ago
        kind of like the idea of a meme (/ˈmiːm/ MEEM) is a meme (/ˈmiːm/ MEEM).
  - TrickyRick 9y ago
    Agreed, it seems like the software approach as well. You wouldn't want a class or a piece of code to be "too big" to fail, you'd refactor it into smaller pieces which can be overviewed more easily.
    - CaptSpify 9y ago
      It's really a basic engineering principle. Smaller and more distributed systems are more reliable.
      - collyw 9y ago
        This sounds like the microkernel / monolith debate. Except monoliths kind of won.
        CaptSpify 9y ago
        It depends on what you want to optimize for. micro is good for some things, mono is good for other things. Neither is inherently "better", it just depends on your goal
  - dalbasal 9y ago
    I don't know what Bernie's specific plans were/are but it's not unusual. At least, at first blush, this is a lot of people's reactions. Too-big-to-fail = danger = make it smaller. Realisitically, this has not been the solution implmented (in the EU & US, at least). In the EU, it is even more crucial as the "solutions" to this problem are applied to state finances as well as financial institutions. In terms of policies, there are two competing approaches: (1) Reduce the size of "too-big-to-fail" institutions. (2) Regulate them more heavily (or some other strategy) so that they will not fail. In the EU, this is being applied to states, not just financial institutions. Rules that (supposedly) reduce catastrophic risk. Almost all seripous policy proposals are in the no. 2 category. Tighten regulation, reduce the risk of failure. Tighter regulation lends to stronger incumbents and larger average company size so by doing 2, you are probably doing the opposite of 1. As I said, I don't know what Bernie's proposal is or how mature it is as a policy (as opposed to a politician statement). It would be notable if a left wing politician propsed loosening bank regulations, though definitely not impossible or unreasonable.
    - ktRolster 9y ago
      Paul Volcker's solution was that any time a bank is so big that we need to bail it out to avoid systemic problems, then the bank should be broken up. We can't always see all the problems in advance, but we can break them up in the aftermath.
  - awkwarddaturtle 9y ago
    > I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist. That seems like the most reasonable response. And yet, since the great recession, our policy has been "make 'too big to fail' even bigger". The problem is that the banks have become too powerful for anyone to challenge. A Teddy Roosevelt type of political leader can't exist today.
    - adventured 9y ago
      Corporate entities were radically more powerful in Teddy Roosevelt's day, far beyond what they are today. They were almost entirely unchained in regards to economic power, whereas today there is hyper regulation (tens of thousands of pages of it, including direct Fed control over the banking system). Standard Oil was as powerful as the US Government in that era, as were the railroads. JP Morgan was far more powerful than the US Treasury. Cornelius Vanderbilt - pre Roosevelt - all by himself had greater financial capability at his peak than the US Government at the time. The difference, is back then there was wide-spread and growing fear of the combinations and would-be monopolies. Today, Americans are relatively unconcerned by Microsoft (desktop), Google (search, android), Walmart & Amazon (retail), Intel (microprocessors), Facebook (social), Cisco, Boeing, etc.
      - pyroinferno 9y ago
        Why leave it at American companies? The Dutch East India Company was far more powerful and bigger than any company seen today.
        bb611 9y ago
        Beyond that, it was more powerful than many (maybe most) modern governments. It was an international power unto itself.
    - tdb7893 9y ago
      Here is an smbc comic making fun of this phenomenon. http://www.smbc-comics.com/index.php?id=3794 http://www.smbc-comics.com/index.php?id=3794
- peteretep 9y ago
  If you wanted to blow something up to make the west suffer, an AWS datacenter would probably be a pretty good target. I wonder at what point that becomes a legitimate national security concern, and the government steps in to provide protection.
  - mediascreen 9y ago
    Wouldn't you have to blow up at least all the datacenters in a region to make an impact?
    - cm2187 9y ago
      All you need to blow is a few cables.
      - cachvico 9y ago
        Then all that's needed to repair is a few cables.
      - mediascreen 9y ago
        Well, you need to disable redundant ISP or power cables to all datacenters in a region. and that would probably be pretty easy to recover from in a day or two. I imagine Amazon has some on site security measures as well. It would be interesting to see how important services would cope with their main region going down for more than a few days.
        cm2187 9y ago
        Not that I think it is wise to discuss optimising a terrorist activity on a public forum, though it is interesting from a threat analysis point of view. You don't need to severe both power and data cables. For power the datacentre should be able to cope for a while, particularly if it has access to fuel deliveries. Data cable should be both easier to severe and more difficult to recover from (not the least to identify which cable has been cut where). Most of the infrastructure of a country in term of cables run along railway tracks, sewers, etc. This means thousands of kilometers of cable even for a small country, and it is impossible to secure everything. It is impossible to severe everything either but you don't need to severe every single cable. As long as you severe enough of the backbone, the other cables will be overloaded. So I don't think it would take that much effort for a network of terrorists to create havoc in the communications of a country for at least a few days to a week. And the consequences for the economy can be pretty dire. We have seen with BA what happens when their datacentre goes offline. Their all fleet is grounded. I imagine the consequences of a country-wide outage could be pretty dramatic. Unlikely anyone would die but you could really dent the GDP.
barsonme 9y ago
Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.
- tylersmith 9y ago
  AWS provides a lot of features that are exclusive to their platform and can't be drop-in replaced on other providers like Azure of GCE. ELB, EFS, S3, ASGs, etc. They'd need to be replaced at the application level for other platforms. That could be a huge commitment for a decent sized system.
  - conradk 9y ago
    I don't know about ELB, EFS and ASG but: - S3 has a public protocol and many 3rd party providers support it (OpenIO, Scality, Ceph, Minio, etc), - EFS could be replaced with something like DRDB or GlusterFS, or DigitalOcean's block storage or Google Cloud's networked disks. - ELB could be replaced easily with similar services from other providers [1] if you use Kubernetes (I don't know if all have a LoadBalancer type though) I would be more concerned about firewall/vpc rules, because I have no idea how those could be migrated without risk of forgetting some. Lock-in seems not that high in the end though and even less so if you use an open source container orchestration stack because they abstract most of these things away. [1] https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#external-load-balancer-providers https://kubernetes.io/docs/tasks/access-application-cluster/...
    - Matt3o12_ 9y ago
      One good way is to have automated tests which make sure that those rules actually work. What if all your AWS rules suddenly get deleted. How are you supposed to know if you have not forgotten any.
      - 9y ago
cjsuk 9y ago
This does worry me. If there is a shortage of resources suddenly or a DC fire that takes out a region, then what? We have contingency against this via our own infrastructure but I worry about organisations who don't have any.
- provost 9y ago
  > Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones. Amazon EC2 provides you the ability to place resources, such as instances, and data in multiple locations. Resources aren't replicated across regions unless you do so specifically. Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
- kondro 9y ago
  One region isn't going to be effected by fire. And AWS have dozens of regions. They're even managed as separate units by separate people. You'll notice there's never been a large, multi-region outage of AWS.
  - cjsuk 9y ago
    Yet. Some of the traditional apps we host are vulnerable hypervisor failure be that rack, DC or region.
    - kondro 9y ago
      Hardware always fails. That's why AWS has so many availability zones, regions and services that let you take easy advantage of HA across them.
- coldtea 9y ago
  >This does worry me. If there is a shortage of resources suddenly or a DC fire that takes out a region, then what? Then some businesses will be out for a few hours / days. No big deal. From WWII to 9/11 to Katrina (and whatever regional stuff we have), we have been through much worse than that in modern history.
smegel 9y ago
Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).
- HatchedLake721 9y ago
  No. Hence them rolling out new features region by region.
- dmurray 9y ago
  A major solar flare and coronal mass ejection? It wouldn't just be Amazon that was affected, though.
- dijit 9y ago
  Unless you work at amazon, you can't know that. It looks very separated on the outside, but I've worked in so many companies that have appeared incredibly competent externally but have "snowflake" servers which keep things ticking over- Given Bezos treatment of workers I have absolutely no confidence that everything is as cleanly engineered as they claim.
- blazespin 9y ago
  Yes, there are ways to bring down all of their arch at once, but you'd have to get through a lot of barriers to do it.
- mrep 9y ago
  That would go against a core principle at aws which is to have every region completely isolated. Also, deployments are designed to be exponential and no region should ever have a cross region dependency.
- askvictor 9y ago
  A cascading electrical grid failure? I don't know if there are any interconnects between the regions with the DC's, but if there were that might be a concern. Though at that stage, presumably most of the US is without power, hence not so much need for AWS.
fovc 9y ago
I think about this problem every now and then for my own business, but not sure what the right answer is. Supporting multiple clouds requires more involved management of some pieces of infrastructure (e.g., DNS + healthchecks, DB replication), which introduces another point of failure. How do people who need to have more nines of availability manage this issue with cloud providers? (EC2 and RDS promise 3.5 nines per AZ, but I imagine outages are somewhat correlated across zones)
- sokoloff 9y ago
  Very, very few businesses should be architecting to ensure higher than 99.95% availability, IMO. (Less than 4.5 hours of downtime per year.) Users are patient enough to give you a pass if you're down that amount (especially if you're down that amount while 1/3rd of the internet is also down). Our largest e-commerce retail site does over $1BB/yr in fairly high-margin sales and still targets "only" 99.95% availability (generally it exceeds that with actual results, but we don't target higher than that). It's a hybrid of on-prem and cloud services backing that, migrating towards the cloud, but will never be 100% cloud as we own and run factories with on-prem equipment. (I know you asked "how" and I answered "whether", but I thought it relevant.)
- dastbe 9y ago
  for people who need more 9s of availability on a single cloud provider, you have to start going multi-region. aws takes region isolation/independence very seriously, and along with geographic independence gives you effectively two entirely independent clouds which just so happen to have the exact same APIs. Some of the (really great) Netflix blog posts[0] have talked about multi-region services. If you do go multi-cloud, I would be wary of picking regions that are located very close to each other. While you'll obviously get independent code and (likely) independent deployments, you're still susceptible to issues correlated with the physical location. [0] https://medium.com/netflix-techblog/global-cloud-active-active-and-beyond-a0fdfa2c3a45 https://medium.com/netflix-techblog/global-cloud-active-acti...
jondubois 9y ago
That's why I think containerization and orchestration will be useful; open source orchestrators can standardize the infrastructure and make switching seamless. That way the infrastructure remains a commodity.
- lukeholder 9y ago
  Except you can't containerize the huge amounts of data you are storing can you?
blazespin 9y ago
The solution is pretty simple, AWS/Azure need to provide on premise versions of their cloud.. You'd probably get stuck with a particular version, but better than nothing.
- arethuza 9y ago
  That's pretty much what Azure Stack is: https://azure.microsoft.com/en-gb/overview/azure-stack/ https://azure.microsoft.com/en-gb/overview/azure-stack/ There might well be a commercial niche for providing Azure Stack hosting in non-Microsoft data centers.
  - bonesss 9y ago
    I think there is a massive market for 100% cloud-compatible local deployments. In my personal experience every .Net shop I've seen would love to be incorporating more Azure goodness locally, but can't as they're cloud specific techs which bump into the realities of deployment and maintenance. Personally, I think MS crapped the bed a little by taking Azure Stack off of commodity hardware and onto a combined hardware/software solution. Being able to deploy Azure-compatible solutions piece-meal locally would be a massive boon to governments, healthcare operations, and anyone working on a more thorough migration to the cloud. Most of the EU, for example, has privacy regulation that makes cloud hosting impossible in some situations. Having a 'local Azure' would make it highly reasonable have all apps architected around Azures components and technology. Without the local deployment though you're kinda stuck with each foot in a different canoe... Hybrid infrastructures are highly favorable to DevOps and multi-party development scenarios.
    - Delphiza 9y ago
      From Scott Guthrie "“So if the performance is dropping, do you call the server manufacturer, do you call the networking manufacturer, do you call the load balancer manufacturer, do you call the storage manufacturer? They typically point the finger at the other guy and you spend weeks and months trying to debug and get your cloud to work." https://www.theregister.co.uk/2017/07/10/interview_with_microsofts_scott_guthrie/?page=2 https://www.theregister.co.uk/2017/07/10/interview_with_micr... We can all relate to that. A "cloud" is sufficiently complex that vendor blaming is an almost guaranteed outcome.
cm2187 9y ago
What would be great is the equivalent of the ACME protocol for cloud service providers. That will take a while and shouldn't happen until the offering matures and stabilises. But in an ideal world you wouldn't tie your application to a specific cloud provider. You should be able to lift and shift to another provider. Which I think is a merit of using VMs as opposed to individual services.
- unixhero 9y ago
  ACME protocol?
  - cm2187 9y ago
    Developped by let's encrypt, which helps solving the too big to fail problem with CA. When CA adopted it (which looks like it may happen), you will have a common protocol to create and renew certificates across CA.
- gaius 9y ago
  But in an ideal world you wouldn't tie your application to a specific cloud provider. You can do that easily if you just treat clouds merely as hosted hypervisors and think entirely in terms of VMDKs. But this doesn't make commercial sense to do at least in the short term - you need to utilise the layered services you are paying for anyway or you might as well just run your own DC.
  - icebraining 9y ago
    It still makes sense for its elastic properties (from which EC2 got its name). You can't rent half a DC for an hour, but you can spawn generic instances from VMDKs on different providers with a fairly small abstraction layer.
    - gaius 9y ago
      Your data still needs to live somewhere and giant VMDKs being copied around aren't a reasonable solution, I'd argue.
nhumrich 9y ago
For articles where the headline is a question, the answer is always "no".
sharemywin 9y ago
Hasn't anyone heard of disaster recover plans? I used to work at a medium sized insurance company and every year we had a project to update our disaster recovery plans. Including our main inhouse datacenter going down. If it was a critical system you'd better have a plan to get it back up in like 4 hours. and those were business critical we didn't have any life critical systems.
- YawningAngel 9y ago
  What's the disaster plan for "DynamoDB doesn't exist any more"? There is literally nothing else like it in the world. I don't know of an idiot proof queue system that can handle the scales SQS can take either.
  - darkr 9y ago
    Cassandra? Rabbit?
AmIFirstToThink 9y ago
If your architecture means your system goes down if AWS is down, then the question becomes can you replace AWS with something better that you can build, have means to build, have time to build, can keep running, can get enough momentum in term of sheer size of customer base to fund the upkeep of the platform? If you can't build/run a better AWS replacement then it's a mute point, isn't it? Then the question turns into if you can't build better AWS, can you architect your application to handle AWS failures? AWS itself lets you handle many kind of failures at AZ/DC level. Are you using that? For global AWS outages, can you have skeleton, survival critical system running on GCP or Azure? Have you thought about outages that would be out of your control and out of AWS's control e.g. malware, DDoS, DNS, ISP, Windows/Android/iOS/Chrome/Edge zero day? How are you going to handle outages due to those issues? If you are prepared to handle outages (communication, self-preservation, degraded mode, offline mode) then can a serious AWS outage be managed just like those outages?
- savoytruffle 9y ago
  irrelevant points are "moot", not "mute"
  - c22 9y ago
    Moot points aren't really irrelevant, on the contrary, they're perhaps the most relevant as non moot points are already settled.
    - falsedan 9y ago
      https://brians.wsu.edu/2016/05/25/mute-point/ https://brians.wsu.edu/2016/05/25/mute-point/ Woah, Prof Brians updated his layout
  - darkr 9y ago
    I think you mean "moo". It's like a cow's opinion, you know, it just doesn't matter. It's "moo".
    - 9y ago
jpalomaki 9y ago
This goes to beyond having a plan-B for hosting your own stuff somewhere else. Think about all the 3rd party services you are depending on. Then think about how many dependencies those services have. How many trace back to Amazon on some level? The connections that could cause problems may not be obvious. For example network provider running into trouble because a ticketing or monitoring system that depends Amazon does not work. Hardware supplier not being able to ship spare parts for your on-premise SAN because logistics company runs into trouble due to issues at Amazon.
martyvis 9y ago
It took me three reads of the first couple of paragraphs to realise that "snowball" and "snowmobile" were actually hardware products that you can touch. Tech news publishers need to do a jargon check and use appropriate punctuation, formatting or something to call out terms that 90% of readers would not have come accross
- cdolan 9y ago
  Maybe its because I saw your comment before reading, but I had no problem understanding the first few paragraphs. The author states that a "snowball" is a grey suitcase with 50tb of HDD space inside, and a "snowmobile" is a massive 18 wheeler with what I would assume is petabytes of storage.
  - martyvis 9y ago
    It's probably because it's 5 in the morning here :-) But looking at Amazon's own references to the appliances, they always capitalise the name. I guess what I can only assume was intentional obscuring what are probably trademarks made it read poorly to me.
- pavel_lishin 9y ago
  Really? It's explicitly stated in the very first paragraph, in the second sentence: > Not the lumps of mush and ice that children chuck at each other, but Amazon’s portable information storage devices, big grey suitcases that hold huge amounts of data. Capitalizing it might have helped, though.
amerine 9y ago
No. https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
nogbit 9y ago
Yes and no. By design it's not big, it just seems big. With relative RPO and RTO anyone can failover to other regions. And if you aren't leveraging multiple AZ's within a single region you need to rethink how you are using AWS. The very nature of AWS requires Amazon to build in capabilities to handle failover. But, as they say at Amazon, "everything fails, always".
galkk 9y ago
When I was working as contractor for one of big banks, which dev was concentrated on Canary Wharf, they weren't able to successfully complete disaster recovery testing on their primary database cluster for 2 years in a row, I just don't remember, was is department-wide or bank-wide. Basically, each 6 months DR testing was failing and it was accepted as harsh reality. After seeing how they're working inside, I don't think that moving their infrastructure to AWS/Azure/Google is worst that could happen. disc: Currently working at Amazon, but not at AWS.
- Kenji 9y ago
  Why did they not redo the DR testing until it worked? Normally you iterate tests and bugfixes over and over until it works. Otherwise, what's the point of the test? Being confident that your stuff does not work at all?
  - galkk 9y ago
    It was bank-wide activity with defined schedule etc.
zeep 9y ago
If Amazon's cloud service would disappear today, it would be a chaos for a week or two but most people should recover (as long as they have backups).
- pavel_lishin 9y ago
  I'd wager most peoples' database backups live in AWS as well. Plus, some people have huge, huge datasets. It could easily take weeks to migrate to, say, GCE, or to your own hosted servers. In the latter case, it would also necessitate a pretty large up-front investment.
acd 9y ago
Cloud services are concentrated by nature built with the same cloned DNA. Of course that is a systematic risk with so much it concentrated to fewer physical locations running on the same code. Think Cloned bananas vs fingers disease but computers. http://www.bbc.com/news/uk-england-35131751 http://www.bbc.com/news/uk-england-35131751
jriot 9y ago
Nothing is too big to fail. Society needs to be able to adapt and maintain a level of patience during transition times i.e., be patient when Amazon's cloud fails to a new tool.
forkLding 9y ago
Personally as a dev, I find AWS's service in the middle of Paypal (shit, not sure why they're popular) to Stripe (Damn that was fast and easy) seeing as I used them both. Their support is alright although you often have to pay for it but AWS docs are atrocious and remind me of university textbooks written by professors who like creating pseudo-scientific-sounding jargon which mixed with their huge array of features is quite un-comforting to use for even people with intermediate AWS experience (built some apps with AWS before kind of people). I can see that there could be more specialized services like Firebase (which is built on Google Cloud) that should be built on AWS for the users. Firebase is a breeze to use and very responsive and I've used it to build real-time chat apps in a couple days.
omi 9y ago
People (junior devs and designers mostly) seem to grossly underestimate the risk of single point of failure. Yeah, yeah distributed, I know... but more of a monoculture problem: most-successful-strain-of-X.

182 points by azureel 9y ago

This is (I was surprised) a pretty good article. Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.

pferde 9y ago
> Financial services are regulated and based on recent experience, they're concerned with systemic risk. Most industries do not have anyone responsible for worrying about this kind of thing. I'd say that most industries do not have anyone responsible for worrying about it high enough in the management chain.
- deleted 9y ago
  [deleted]
tyfon 9y ago
I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist. That should be the priority.
- 65827 9y ago
  Why is every arrangement of characters now a "meme". That word has moved beyond devoid of meaning, at this point it's like a black hole of nothingness of a word.
  - TrickyRick 9y ago
    > A meme (/ˈmiːm/ MEEM) is an idea, behavior, or style that spreads from person to person within a culture. Seems like it's fitting here, does it not? Certain banks being "too big to fail" is an idea passed among persons within our culture.
    - sharemywin 9y ago
      kind of like the idea of a meme (/ˈmiːm/ MEEM) is a meme (/ˈmiːm/ MEEM).
- TrickyRick 9y ago
  Agreed, it seems like the software approach as well. You wouldn't want a class or a piece of code to be "too big" to fail, you'd refactor it into smaller pieces which can be overviewed more easily.
  - CaptSpify 9y ago
    It's really a basic engineering principle. Smaller and more distributed systems are more reliable.
    - collyw 9y ago
      This sounds like the microkernel / monolith debate. Except monoliths kind of won.
      - CaptSpify 9y ago
        It depends on what you want to optimize for. micro is good for some things, mono is good for other things. Neither is inherently "better", it just depends on your goal
- dalbasal 9y ago
  I don't know what Bernie's specific plans were/are but it's not unusual. At least, at first blush, this is a lot of people's reactions. Too-big-to-fail = danger = make it smaller. Realisitically, this has not been the solution implmented (in the EU & US, at least). In the EU, it is even more crucial as the "solutions" to this problem are applied to state finances as well as financial institutions. In terms of policies, there are two competing approaches: (1) Reduce the size of "too-big-to-fail" institutions. (2) Regulate them more heavily (or some other strategy) so that they will not fail. In the EU, this is being applied to states, not just financial institutions. Rules that (supposedly) reduce catastrophic risk. Almost all seripous policy proposals are in the no. 2 category. Tighten regulation, reduce the risk of failure. Tighter regulation lends to stronger incumbents and larger average company size so by doing 2, you are probably doing the opposite of 1. As I said, I don't know what Bernie's proposal is or how mature it is as a policy (as opposed to a politician statement). It would be notable if a left wing politician propsed loosening bank regulations, though definitely not impossible or unreasonable.
  - ktRolster 9y ago
    Paul Volcker's solution was that any time a bank is so big that we need to bail it out to avoid systemic problems, then the bank should be broken up. We can't always see all the problems in advance, but we can break them up in the aftermath.
- awkwarddaturtle 9y ago
  > I really hate the "too big to fail" meme and I strongly agree with Bernie in that if you are too big to fail you are too big to exist. That seems like the most reasonable response. And yet, since the great recession, our policy has been "make 'too big to fail' even bigger". The problem is that the banks have become too powerful for anyone to challenge. A Teddy Roosevelt type of political leader can't exist today.
  - adventured 9y ago
    Corporate entities were radically more powerful in Teddy Roosevelt's day, far beyond what they are today. They were almost entirely unchained in regards to economic power, whereas today there is hyper regulation (tens of thousands of pages of it, including direct Fed control over the banking system). Standard Oil was as powerful as the US Government in that era, as were the railroads. JP Morgan was far more powerful than the US Treasury. Cornelius Vanderbilt - pre Roosevelt - all by himself had greater financial capability at his peak than the US Government at the time. The difference, is back then there was wide-spread and growing fear of the combinations and would-be monopolies. Today, Americans are relatively unconcerned by Microsoft (desktop), Google (search, android), Walmart & Amazon (retail), Intel (microprocessors), Facebook (social), Cisco, Boeing, etc.
    - pyroinferno 9y ago
      Why leave it at American companies? The Dutch East India Company was far more powerful and bigger than any company seen today.
      - bb611 9y ago
        Beyond that, it was more powerful than many (maybe most) modern governments. It was an international power unto itself.
  - tdb7893 9y ago
    Here is an smbc comic making fun of this phenomenon. http://www.smbc-comics.com/index.php?id=3794 http://www.smbc-comics.com/index.php?id=3794
peteretep 9y ago
If you wanted to blow something up to make the west suffer, an AWS datacenter would probably be a pretty good target. I wonder at what point that becomes a legitimate national security concern, and the government steps in to provide protection.
- mediascreen 9y ago
  Wouldn't you have to blow up at least all the datacenters in a region to make an impact?
  - cm2187 9y ago
    All you need to blow is a few cables.
    - cachvico 9y ago
      Then all that's needed to repair is a few cables.
    - mediascreen 9y ago
      Well, you need to disable redundant ISP or power cables to all datacenters in a region. and that would probably be pretty easy to recover from in a day or two. I imagine Amazon has some on site security measures as well. It would be interesting to see how important services would cope with their main region going down for more than a few days.
      - cm2187 9y ago
        Not that I think it is wise to discuss optimising a terrorist activity on a public forum, though it is interesting from a threat analysis point of view. You don't need to severe both power and data cables. For power the datacentre should be able to cope for a while, particularly if it has access to fuel deliveries. Data cable should be both easier to severe and more difficult to recover from (not the least to identify which cable has been cut where). Most of the infrastructure of a country in term of cables run along railway tracks, sewers, etc. This means thousands of kilometers of cable even for a small country, and it is impossible to secure everything. It is impossible to severe everything either but you don't need to severe every single cable. As long as you severe enough of the backbone, the other cables will be overloaded. So I don't think it would take that much effort for a network of terrorists to create havoc in the communications of a country for at least a few days to a week. And the consequences for the economy can be pretty dire. We have seen with BA what happens when their datacentre goes offline. Their all fleet is grounded. I imagine the consequences of a country-wide outage could be pretty dramatic. Unlikely anyone would die but you could really dent the GDP.

barsonme 9y ago

Even at a smaller scale it is a little nerve-wracking to know be so reliant on one provider. If AWS tanks there's a fair amount of code that'd need to be changed just to switch over to Azure or GCE. Failover with, e.g., email providers is easy enough, but the entire cloud stack (for lack of better terms) is a completely different ballgame.

tylersmith 9y ago
AWS provides a lot of features that are exclusive to their platform and can't be drop-in replaced on other providers like Azure of GCE. ELB, EFS, S3, ASGs, etc. They'd need to be replaced at the application level for other platforms. That could be a huge commitment for a decent sized system.
- conradk 9y ago
  I don't know about ELB, EFS and ASG but: - S3 has a public protocol and many 3rd party providers support it (OpenIO, Scality, Ceph, Minio, etc), - EFS could be replaced with something like DRDB or GlusterFS, or DigitalOcean's block storage or Google Cloud's networked disks. - ELB could be replaced easily with similar services from other providers [1] if you use Kubernetes (I don't know if all have a LoadBalancer type though) I would be more concerned about firewall/vpc rules, because I have no idea how those could be migrated without risk of forgetting some. Lock-in seems not that high in the end though and even less so if you use an open source container orchestration stack because they abstract most of these things away. [1] https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#external-load-balancer-providers https://kubernetes.io/docs/tasks/access-application-cluster/...
  - Matt3o12_ 9y ago
    One good way is to have automated tests which make sure that those rules actually work. What if all your AWS rules suddenly get deleted. How are you supposed to know if you have not forgotten any.
    - 9y ago

cjsuk 9y ago

This does worry me. If there is a shortage of resources suddenly or a DC fire that takes out a region, then what? We have contingency against this via our own infrastructure but I worry about organisations who don't have any.

provost 9y ago
> Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones. Amazon EC2 provides you the ability to place resources, such as instances, and data in multiple locations. Resources aren't replicated across regions unless you do so specifically. Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
kondro 9y ago
One region isn't going to be effected by fire. And AWS have dozens of regions. They're even managed as separate units by separate people. You'll notice there's never been a large, multi-region outage of AWS.
- cjsuk 9y ago
  Yet. Some of the traditional apps we host are vulnerable hypervisor failure be that rack, DC or region.
  - kondro 9y ago
    Hardware always fails. That's why AWS has so many availability zones, regions and services that let you take easy advantage of HA across them.
coldtea 9y ago
>This does worry me. If there is a shortage of resources suddenly or a DC fire that takes out a region, then what? Then some businesses will be out for a few hours / days. No big deal. From WWII to 9/11 to Katrina (and whatever regional stuff we have), we have been through much worse than that in modern history.

smegel 9y ago

Is it possible for AWS to have a multi-region outage - as in is there anything connecting them that could bring them all (or several) down at once? (Apart from the result of a botched patching or update to the core software stack that was done worldwide at the same time and hopefully never happens).

HatchedLake721 9y ago
No. Hence them rolling out new features region by region.
dmurray 9y ago
A major solar flare and coronal mass ejection? It wouldn't just be Amazon that was affected, though.
dijit 9y ago
Unless you work at amazon, you can't know that. It looks very separated on the outside, but I've worked in so many companies that have appeared incredibly competent externally but have "snowflake" servers which keep things ticking over- Given Bezos treatment of workers I have absolutely no confidence that everything is as cleanly engineered as they claim.
blazespin 9y ago
Yes, there are ways to bring down all of their arch at once, but you'd have to get through a lot of barriers to do it.
mrep 9y ago
That would go against a core principle at aws which is to have every region completely isolated. Also, deployments are designed to be exponential and no region should ever have a cross region dependency.
askvictor 9y ago
A cascading electrical grid failure? I don't know if there are any interconnects between the regions with the DC's, but if there were that might be a concern. Though at that stage, presumably most of the US is without power, hence not so much need for AWS.

fovc 9y ago

I think about this problem every now and then for my own business, but not sure what the right answer is. Supporting multiple clouds requires more involved management of some pieces of infrastructure (e.g., DNS + healthchecks, DB replication), which introduces another point of failure. How do people who need to have more nines of availability manage this issue with cloud providers? (EC2 and RDS promise 3.5 nines per AZ, but I imagine outages are somewhat correlated across zones)

sokoloff 9y ago
Very, very few businesses should be architecting to ensure higher than 99.95% availability, IMO. (Less than 4.5 hours of downtime per year.) Users are patient enough to give you a pass if you're down that amount (especially if you're down that amount while 1/3rd of the internet is also down). Our largest e-commerce retail site does over $1BB/yr in fairly high-margin sales and still targets "only" 99.95% availability (generally it exceeds that with actual results, but we don't target higher than that). It's a hybrid of on-prem and cloud services backing that, migrating towards the cloud, but will never be 100% cloud as we own and run factories with on-prem equipment. (I know you asked "how" and I answered "whether", but I thought it relevant.)
dastbe 9y ago
for people who need more 9s of availability on a single cloud provider, you have to start going multi-region. aws takes region isolation/independence very seriously, and along with geographic independence gives you effectively two entirely independent clouds which just so happen to have the exact same APIs. Some of the (really great) Netflix blog posts[0] have talked about multi-region services. If you do go multi-cloud, I would be wary of picking regions that are located very close to each other. While you'll obviously get independent code and (likely) independent deployments, you're still susceptible to issues correlated with the physical location. [0] https://medium.com/netflix-techblog/global-cloud-active-active-and-beyond-a0fdfa2c3a45 https://medium.com/netflix-techblog/global-cloud-active-acti...

jondubois 9y ago

That's why I think containerization and orchestration will be useful; open source orchestrators can standardize the infrastructure and make switching seamless. That way the infrastructure remains a commodity.

lukeholder 9y ago
Except you can't containerize the huge amounts of data you are storing can you?

blazespin 9y ago

The solution is pretty simple, AWS/Azure need to provide on premise versions of their cloud.. You'd probably get stuck with a particular version, but better than nothing.

arethuza 9y ago
That's pretty much what Azure Stack is: https://azure.microsoft.com/en-gb/overview/azure-stack/ https://azure.microsoft.com/en-gb/overview/azure-stack/ There might well be a commercial niche for providing Azure Stack hosting in non-Microsoft data centers.
- bonesss 9y ago
  I think there is a massive market for 100% cloud-compatible local deployments. In my personal experience every .Net shop I've seen would love to be incorporating more Azure goodness locally, but can't as they're cloud specific techs which bump into the realities of deployment and maintenance. Personally, I think MS crapped the bed a little by taking Azure Stack off of commodity hardware and onto a combined hardware/software solution. Being able to deploy Azure-compatible solutions piece-meal locally would be a massive boon to governments, healthcare operations, and anyone working on a more thorough migration to the cloud. Most of the EU, for example, has privacy regulation that makes cloud hosting impossible in some situations. Having a 'local Azure' would make it highly reasonable have all apps architected around Azures components and technology. Without the local deployment though you're kinda stuck with each foot in a different canoe... Hybrid infrastructures are highly favorable to DevOps and multi-party development scenarios.
  - Delphiza 9y ago
    From Scott Guthrie "“So if the performance is dropping, do you call the server manufacturer, do you call the networking manufacturer, do you call the load balancer manufacturer, do you call the storage manufacturer? They typically point the finger at the other guy and you spend weeks and months trying to debug and get your cloud to work." https://www.theregister.co.uk/2017/07/10/interview_with_microsofts_scott_guthrie/?page=2 https://www.theregister.co.uk/2017/07/10/interview_with_micr... We can all relate to that. A "cloud" is sufficiently complex that vendor blaming is an almost guaranteed outcome.

cm2187 9y ago

What would be great is the equivalent of the ACME protocol for cloud service providers. That will take a while and shouldn't happen until the offering matures and stabilises. But in an ideal world you wouldn't tie your application to a specific cloud provider. You should be able to lift and shift to another provider. Which I think is a merit of using VMs as opposed to individual services.

unixhero 9y ago
ACME protocol?
- cm2187 9y ago
  Developped by let's encrypt, which helps solving the too big to fail problem with CA. When CA adopted it (which looks like it may happen), you will have a common protocol to create and renew certificates across CA.
gaius 9y ago
But in an ideal world you wouldn't tie your application to a specific cloud provider. You can do that easily if you just treat clouds merely as hosted hypervisors and think entirely in terms of VMDKs. But this doesn't make commercial sense to do at least in the short term - you need to utilise the layered services you are paying for anyway or you might as well just run your own DC.
- icebraining 9y ago
  It still makes sense for its elastic properties (from which EC2 got its name). You can't rent half a DC for an hour, but you can spawn generic instances from VMDKs on different providers with a fairly small abstraction layer.
  - gaius 9y ago
    Your data still needs to live somewhere and giant VMDKs being copied around aren't a reasonable solution, I'd argue.

nhumrich 9y ago

For articles where the headline is a question, the answer is always "no".

sharemywin 9y ago

Hasn't anyone heard of disaster recover plans? I used to work at a medium sized insurance company and every year we had a project to update our disaster recovery plans. Including our main inhouse datacenter going down. If it was a critical system you'd better have a plan to get it back up in like 4 hours. and those were business critical we didn't have any life critical systems.

YawningAngel 9y ago
What's the disaster plan for "DynamoDB doesn't exist any more"? There is literally nothing else like it in the world. I don't know of an idiot proof queue system that can handle the scales SQS can take either.
- darkr 9y ago
  Cassandra? Rabbit?

AmIFirstToThink 9y ago

If your architecture means your system goes down if AWS is down, then the question becomes can you replace AWS with something better that you can build, have means to build, have time to build, can keep running, can get enough momentum in term of sheer size of customer base to fund the upkeep of the platform? If you can't build/run a better AWS replacement then it's a mute point, isn't it? Then the question turns into if you can't build better AWS, can you architect your application to handle AWS failures? AWS itself lets you handle many kind of failures at AZ/DC level. Are you using that? For global AWS outages, can you have skeleton, survival critical system running on GCP or Azure? Have you thought about outages that would be out of your control and out of AWS's control e.g. malware, DDoS, DNS, ISP, Windows/Android/iOS/Chrome/Edge zero day? How are you going to handle outages due to those issues? If you are prepared to handle outages (communication, self-preservation, degraded mode, offline mode) then can a serious AWS outage be managed just like those outages?

savoytruffle 9y ago
irrelevant points are "moot", not "mute"
- c22 9y ago
  Moot points aren't really irrelevant, on the contrary, they're perhaps the most relevant as non moot points are already settled.
  - falsedan 9y ago
    https://brians.wsu.edu/2016/05/25/mute-point/ https://brians.wsu.edu/2016/05/25/mute-point/ Woah, Prof Brians updated his layout
- darkr 9y ago
  I think you mean "moo". It's like a cow's opinion, you know, it just doesn't matter. It's "moo".
  - 9y ago

jpalomaki 9y ago

This goes to beyond having a plan-B for hosting your own stuff somewhere else. Think about all the 3rd party services you are depending on. Then think about how many dependencies those services have. How many trace back to Amazon on some level? The connections that could cause problems may not be obvious. For example network provider running into trouble because a ticketing or monitoring system that depends Amazon does not work. Hardware supplier not being able to ship spare parts for your on-premise SAN because logistics company runs into trouble due to issues at Amazon.

martyvis 9y ago

It took me three reads of the first couple of paragraphs to realise that "snowball" and "snowmobile" were actually hardware products that you can touch. Tech news publishers need to do a jargon check and use appropriate punctuation, formatting or something to call out terms that 90% of readers would not have come accross

cdolan 9y ago
Maybe its because I saw your comment before reading, but I had no problem understanding the first few paragraphs. The author states that a "snowball" is a grey suitcase with 50tb of HDD space inside, and a "snowmobile" is a massive 18 wheeler with what I would assume is petabytes of storage.
- martyvis 9y ago
  It's probably because it's 5 in the morning here :-) But looking at Amazon's own references to the appliances, they always capitalise the name. I guess what I can only assume was intentional obscuring what are probably trademarks made it read poorly to me.
pavel_lishin 9y ago
Really? It's explicitly stated in the very first paragraph, in the second sentence: > Not the lumps of mush and ice that children chuck at each other, but Amazon’s portable information storage devices, big grey suitcases that hold huge amounts of data. Capitalizing it might have helped, though.

amerine 9y ago

No. https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

nogbit 9y ago

Yes and no. By design it's not big, it just seems big. With relative RPO and RTO anyone can failover to other regions. And if you aren't leveraging multiple AZ's within a single region you need to rethink how you are using AWS. The very nature of AWS requires Amazon to build in capabilities to handle failover. But, as they say at Amazon, "everything fails, always".

galkk 9y ago

When I was working as contractor for one of big banks, which dev was concentrated on Canary Wharf, they weren't able to successfully complete disaster recovery testing on their primary database cluster for 2 years in a row, I just don't remember, was is department-wide or bank-wide. Basically, each 6 months DR testing was failing and it was accepted as harsh reality. After seeing how they're working inside, I don't think that moving their infrastructure to AWS/Azure/Google is worst that could happen. disc: Currently working at Amazon, but not at AWS.

Kenji 9y ago
Why did they not redo the DR testing until it worked? Normally you iterate tests and bugfixes over and over until it works. Otherwise, what's the point of the test? Being confident that your stuff does not work at all?
- galkk 9y ago
  It was bank-wide activity with defined schedule etc.

zeep 9y ago

If Amazon's cloud service would disappear today, it would be a chaos for a week or two but most people should recover (as long as they have backups).

pavel_lishin 9y ago
I'd wager most peoples' database backups live in AWS as well. Plus, some people have huge, huge datasets. It could easily take weeks to migrate to, say, GCE, or to your own hosted servers. In the latter case, it would also necessitate a pretty large up-front investment.

acd 9y ago

Cloud services are concentrated by nature built with the same cloned DNA. Of course that is a systematic risk with so much it concentrated to fewer physical locations running on the same code. Think Cloned bananas vs fingers disease but computers. http://www.bbc.com/news/uk-england-35131751 http://www.bbc.com/news/uk-england-35131751

jriot 9y ago

Nothing is too big to fail. Society needs to be able to adapt and maintain a level of patience during transition times i.e., be patient when Amazon's cloud fails to a new tool.

forkLding 9y ago

Personally as a dev, I find AWS's service in the middle of Paypal (shit, not sure why they're popular) to Stripe (Damn that was fast and easy) seeing as I used them both. Their support is alright although you often have to pay for it but AWS docs are atrocious and remind me of university textbooks written by professors who like creating pseudo-scientific-sounding jargon which mixed with their huge array of features is quite un-comforting to use for even people with intermediate AWS experience (built some apps with AWS before kind of people). I can see that there could be more specialized services like Firebase (which is built on Google Cloud) that should be built on AWS for the users. Firebase is a breeze to use and very responsive and I've used it to build real-time chat apps in a couple days.

omi 9y ago

People (junior devs and designers mostly) seem to grossly underestimate the risk of single point of failure. Yeah, yeah distributed, I know... but more of a monoculture problem: most-successful-strain-of-X.