8 ms·
This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsi
by bob1029 6y ago
This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible.
We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and S3. I.e. the actual compute/memory/storage/network capacity, and the mapping of the outside world to it. Because we are a software company, we write most of our own software. There is no value to our customers in us stringing together a pile of someone else's products, especially in a way that we cannot guarantee will be sustainable for >5 years. We cannot afford to constantly rework completed product installations.
We strongly feel that any deeper buy-in with 3rd party technology vendors would compromise our agility and put us at their total mercy. Where we are currently positioned in the lock-in game, we could pull the ripcord and be sitting in a private datacenter within a week. All we need to do is move VMs, domain registrations and DNS nameservers if we want to kill the AWS bill.
I feel for those who are up to their eyeballs in cloud infrastructure. Perhaps you made your own bed, but you shouldn't have to suffer in it. These are very complex decisions. We didn't get it right at first either. Maybe consider pleading with your executive management for mercy now. Perhaps you get a shot at a complete redo before it all comes crashing down. We certainly did. It's amazing what can happen if you have the guts to own up to bad choices and start an honest conversation.
I would also be interested to hear the other side of the coin. Who out there is using 20+ AWS/Azure/GCP products to back a single business app and is having a fantastic time of it?
- rvanmil 6y agoWe carefully select and use PaaS and managed cloud services to construct our infrastructure with. This allows us to maximize our focus on what our customers are paying for: creating software for them which will typically be in use for 5+ years. We spend close to zero time on infrastructure maintenance and management, we pay others to do this for us, cheaper and more reliable. Having to swap out one service for another hasn't given us any trouble or unreasonable costs yet in the past 5 years. Unlike the article is trying to convince us of, it has massively reduced complexity for us.
- bane 6y agoI've worked with a number of teams over the last few years who use AWS and I'd say from top to bottom they all build their strategy more or less the same way: 0. Whatever is the minimum needed to get a VPC stood up. 1. EC2 as 90%+ of whatever they're doing 2. S3 for storing lots of stuff and/or crossing VPC boundaries for data ingress/egress (like seriously, S3 seems to be used more as an alternative to SFTP than for anything else). This makes up usually the rest of the thinking. 3. Maybe one other technology that's usually from the set of {Lambda, Batch, Redshift, SQS} but rarely any combination of two or more of those. And that's it. I know there are teams that go all in. But for the dozen or teams I've personally interacted with this is it. The rest of the stack is usually something stuffed into an EC2 instance instead of using an AWS version and it comes down to one thing: the difficulties in estimating pricing for those pieces. EC2 instances are drop-dead simple to price estimate forward 6 months, 12 months or longer. Amazon is probably leaving billions on the table every year because nobody can figure out how to price things so their department can make their yearly budget requests. The one time somebody tries to use some managed service that goes overbudget by 3000%, and the after action figures out that it would have been within the budget by using <open source technology> in EC2, they just do that instead -- even though it increases the staff cost and maintenance complexity. In fact just this past week a team was looking at using SageMaker in an effort to go all "cloud native", took one look at the pricing sheet and noped right back to Jupyter and scikit_learn in a few EC2 instances. An entire different group I'm working with is evaluating cloud management tools and most of them just simplify provisioning EC2 instances and tracking instance costs. They really don't do much for tracking costs from almost any of the other services.
- harikb 6y ago+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. Infinite scalability is also a curse - we had a case where pruning history from an S3 bucket was failing for months and we didn’t know until the storage bill became significant enough to notice. I guess in some ways it is better than being woken up in the middle of night but we wasted millions storing useless data Azure also has similar issues - deleting a VM sometimes doesn’t cleanup dependent resources and it is a mess to find and delete later - only because the dependent resources are deliberately not named with a matching tag.
- maria_weber23 6y agoI second that. It's not only that you make yourself completely intertwined with a Cloud by using more than fundamental services. The costs of lambda or even DDB are IMMENSE. These only pay off for services that have a high return per request. I.e. if you get a lot of value out of lambda calls, sure, use them. But for anything high-frequency that earns you little to nothing on its own, forget about it. Generally all your critical infrastructure should be Cloud independent. That narrows your choices largely to EC2, SQS, perhaps Kinesis, Rout53, and the like. And even there you should implement all your features with two clouds, i.e. Azure and AWS, just to be sure. The good news is also the bad news. There are effectively only two options: Azure or AWS. Google Cloud is a joke. They arbitrary change their prices, terminate existing products, offer zero support. It's just like we have come to love Google. They just don't give a shit about customers. Google only cares about "architecture", i.e. how cool do I feel as engineer having built that service. Customer service is something that Google doesn't seem to understand. So think carefully if you want to buy into their "product". Google, literally, only develops products for their own benefit.
- jbmsf 6y agoI can't agree, at least not in general. The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate and because of the functionality you have to exclude from your systems. I am sure there are scales where you either have such a large engineering budget that you can ignore these costs or where decreasing your cloud spend is the only way to scale your business. But for the average company, I can't see how spending so much on infrastructure (and future optionality) pays off, especially when you could spend on product or marketing or anything else that has a more direct impact on your success.
- jsiepkes 6y ago> But for the average company, I can't see how spending so much on infrastructure (and future optionality) pays off, especially when you could spend on product or marketing or anything else that has a more direct impact on your success. If you change "average company" to "average startup" then your point make sense. But for a normal company not everything needs to make a direct impact on your success. For example guaranteeing long term business continuity is an important factor too.
- jbmsf 6y agoIt's always a trade-off though. You say you write most of your own software, but that's probably not true for, say your OS or programming language, or editors, or a million other things. Cloud software is the same; you might not be producing the most value if you spend your engineering hours (re)creating something you could buy. In my own experience: - AWS SNS and SQS are rock solid and provide excellent foundations for distributed systems. I know I would struggle to create the same level of reliability if I wrote my own publish-subscribe primitives and I've played enough with some of the open source alternatives to know they require operational costs that I don't want to pay. - I use EC2 some of the time (e.g. when I need GPUs), but I prefer to use containers because they offer a superior solution for reproducible installation. I tend to use ECS because I don't want to take on the complexity of K8S and it offers me enough to have reliable, load-balanced services. ECS with Fargate is a great building block for many, run-of-the-mill services (e.g. no GPU, not crazy resource usages). - Lambda is incredibly useful as glue between systems. I use Lambda to connect S3, SES, CloudWatch, and SQS to application code. I've also gone without Lambda on the SQS side and written my framework layers to dispatch messages to application code. This has advantages (e.g. finer-grain backoff control) but isn't worth it for smaller projects. - Secrets manager is a nice foundational component. There are alternatives out there, but it integrates so well with ECS that I rarely consider them. - RDS is terrific. In a past life, I spent time writing database failover logic and it was way too hard to get right consistently. I love not having to think about it. Plus encryption, backup, and monitoring are all batteries included. - VPC networking is essential. I've seen too many setups that just use the default VPC and run an EC2 instance on a public IP. The horror. - I've recently started to appreciate the value of Step Functions. When I write distributed systems, I tend to end up with a number of discrete components that each handle one part of a problem domain. This works, but creates understandability problems. I don't love writing Step Functions using a JSON grammar that isn't easy to test locally, but I find that the visibility they offer in terms of tracing a workflow is very nice. - CloudFront isn't the best CDN, but it is often good enough. I tend to use it for frontend application hosting (along with S3, Route53, and ACM). - CloudWatch is hard to avoid, though I rather dislike it. CloudWatch rules are useful for implementing cron-like triggers and detecting events in AWS systems, for example knowing whether EC2 failed to provision spot capacity. - I have mixed feeling about DynamoDB as well. It offers a nice set of primitives and is often easier to starting use for small projects than something like RDS, but I rarely operate at the scales where it's a better solution than something like RDS PostgreSQL with all the terrific libraries and frameworks that work with it. - At some scale, you want to segregate AWS resources across different accounts, usually with SSO and some level of automated provisioning. You can't escape IAM here and Control Tower is a pretty nice solution element as well. I'm not sure if I'm up to 20 services yet, but it's probably close enough to answer your question. There are better and worse services out there, but you can get a lot of business value by making the right trade-offs, both because you get something that would be hard to build with the same level of reliability and security and because you can spend your time writing software that speaks more directly to product needs. As for "having a fantastic time", YMMV. I am a huge fan of Terraform and tend to enjoy developing at that level. The solutions I've built provide building blocks for development teams who mostly don't have to think about the services.
- jasode 6y ago>I would also be interested to hear the other side of the coin. Who out there is using 20+ AWS/Azure/GCP products to back a single business app and is having a fantastic time of it? Netflix uses a lot of AWS higher-level services beyond the basics of EC2 + S3. Netflix definitely doesn't restrict its use of AWS to only be a "dumb data center". Across various tech presentations by Netflix engineers, I count at least 17 AWS services they use. + EC2, S3, RDS, DynamoDB, EMR, ELB, Redshift, Lambda, Kinesis, VPC, Route 53, CloudTrail, CloudWatch, SQS, SES, ECS, SimpleDB, <probably many more>. I think we can assume they use 20+ AWS services.
- p_l 6y agoCertain services IMHO have to be discounted from this list: - VPC - basic building block for any AWS-based infra that isn't ancient - CloudTrail - only way to get audit logs out of AWS, no matter what you feed them into - CloudWatch - similar with CloudTrail, many things (but not all) will log to CloudWatch, and if you use your own log infra you'll have to pull from it. Also necessary for metrics. - ELB/ELBv2/NLB/ALB - for many reasons they are often the only ways to pull traffic to your services deployed on AWS. Yes, you can sometimes do it another way around, but you have high chances of feeling the pain. My personal typical set for AWS is EC2, RDS, all the VPC/ELB/NLB/ALB stack, Route53, CloudTrail + CloudWatch. S3 and RDS as needed, as both are easily moved elsewhere.
- tidepod12 6y agoI don't think you can discount them like that. Maybe they aren't as front of mind as services like S3, EC2, etc, but if you were to try to rebuild your setup in a personal data center, replacing the capabilities of VPC, IAM, CloudTrail, NAT gateways, ELBs, KMS etc would be a huge effort on your part. The fact that they are "basic building blocks" makes them more important, not less. In a discussion about the complexity of cloud providers versus other setups, that seems especially relevant.
- p_l 6y agoOh, I meant it more in terms of "can you count on them as optional services". Because they aren't optional, and yes, it takes non trivial amount to replicate them... but funnily enough, several of them have to be replicated elsewhere too. NAT gateways usually aren't an issue, KMS for many places can be done relatively quickly with Hashicorp Vault. IAM is a weird case, because unless you're building a cloud for others to use it's not necessarily that important, meanwhile your own authorization framework is necessary even on AWS because you can't just piggy back on IAM (I wish I could).
- deleted 6y ago[deleted]
- mumblemumble 6y agoI recently inherited a product that was developed from the ground up on AWS. It's been a real eye opener. Yes, it absolutely is locked in, and will never run on anything but AWS. That doesn't surprise me. What surprises me is all of the unnecessary complexity. It's one big Rube Goldberg contraption, stringing together different AWS products, with a great deal of "tool in search of problem" syndrome for good measure. I am pretty sure that, in at least a few spots, the glue code used to plug into Amazon XYZ amounted to a greater development and maintenance burden than a homegrown module for solving the same problem would have been. NIH syndrome is certainly not any fun. But IH syndrome seems to be no better.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- bird_monster 6y ago> There is no value to our customers in us stringing together a pile of someone else's products Maybe not your business, but there are many businesses in which this is exactly what happens. Any managed-service is just combining other people's work into a "product" that gets sold to customers. And that's great! AWS has a staggering amount of products, and lots of business don't even want to have to care about AWS. > Who out there is using 20+ AWS/Azure/GCP products to back a single business app and is having a fantastic time of it? Several times. I think cloud products are just tools to get you further along in your business. Most of the tools I use are distributed systems tools, because I don't want to have to own them, and container runtimes/datastores. Every single thing I've ever deployed across AWS/Azure is used as a generic interface that could be replaced relatively easily if necessary, and I've used Terraform to manage my infrastructure creation/deployment process, so that I can swap resources in and out without having to change tech. If, for some reason, Azure Event Hub stopped providing what we needed it for, we could certainly deploy a customized Kafka implementation and have the rest of our code not really know or care, but from when we set out to build our products, that has always been a "If we need to" problem, and we've never needed to.
- notretarded 6y agoI'm in two minds about this (deeper integration with a particular vendor - i.e. "serverless") Reduced time to market is incredibly valuable. Current client base is well in its millions. Ability to test to few and roll out to many instantly is invaluable. You no longer have to hire competent software developers who understand all patterns and practices to make scalable code and infrastructure. Just need them to work on a particular unit or function. The thing which scares me is, some of these companies are decades of years old, hundreds. How long has AWS/GCP/Azure abstractions been around for? How quick are we to graveyard some of these platforms. Quite. A lot quicker than you can lift, shift and rewrite your solution to elsewhere.
- g9yuayon 6y agoSo your company cautiously chooses which services in AWS to use, and sticks to infrastructure offerings for now. Netflix called it "paved path", and it worked really well too for Netflix. Over the years, though, the "paved path" expanded and extended to more services. It's worth noting that EC2 alone is a huge productivity booster, bar none. Nothing beats setting up a cluster of machines, with a few clicks, that auto scales per dynamic scaling policies. In contrast, Uber couldn't do this for at least 5 years, and their docker-based cluster system is a crippled for not supporting the damn persistent volumes. God knows how much productivity was lost because of the bogus reasons Uber had for not going to cloud.
- mycall 6y agoDid you look into multi-cloud solutions like Pulumi or Terraform to abstract your cloud vendor?
- tiew9Vii 6y agoDon't fall for that trap. I have never used Pulumi, I've used Terraform a bit. I like Terraform so this isn't a dig at the tools. Abstracting your cloud provider is similar to abstracting your database. At a high-level they appear to be the same, they do similar things however the are very different when you get to the fine detail. Pulmi / Terraform are useful in the provide a common language / API between cloud providers however you will never just switch cloud provider with changing a few lines of code.
- twistedpair 6y agoIt depends on your company. For new startups, I'd stick with delivering your product over building your own cloud services on top of EC2/S3. At my last startup, the engineering head shared the Not-Invented-Here view. We couldn't use any handy services. We had to run our own Cassandra and everything else services. It was a huge time sink for a small team that didn't deliver differentiating value. At my current startup, we're almost 100% serverless on SaaS providers. We operate a thousand or more nodes with no ops team (a couple engineers know ops, when needed). We leave the complexity of scaling an maintenance to our cloud vendor's autoscaling services. Sure, we could reduce our opex in places by trying to roll our own, but the ability to have the vast majority of engineers delivering customer product, rather than reinventing the wheel is more valuable to us than fear of vendor lock-in. If our cloud vendor wanted to jack rates, we'd take the necessary action, which we've done before when we banged a service in a week that a vendor was going to hold us over a barrel for, then dropped the vendor.
- MaxBarraclough 6y ago> The one time somebody tries to use some managed service that goes overbudget by 3000%, and the after action figures out that it would have been within the budget by using <open source technology> in EC2, they just do that instead This impacts casual dabblers too. More than once I've seen HN comments on how someone is wary to experiment with cloud computing because a single screw-up can lead to an essentially unlimited bill. Judging by HackerNews anecdotes of when this does happen (unexpected overruns of thousands of dollars), there's a reasonable chance Amazon will refund you out of good will, but that's not enough to lay the fears to rest. Linode let you pre-pay, for instance, but (to my knowledge) this isn't an option offered by Amazon, Microsoft, or Google.
- ak217 6y agoYou seem to have made up your mind, but for the benefit of others: yes, we use many AWS products and are having a fantastic time of it. AWS services are more reliable, less buggy and have more stable APIs than any of the alternatives. Specific services that made a difference to us are Fargate, Spot, Batch, SQS/SNS, Lambda, ECR, EFS, ELB/ACM, CloudWatch Logs, IAM, Aurora RDS, SSM/Secrets Manager, and Step Functions - in addition to EC2/VPC/EBS, Route53 and S3 that you already listed. Each of these services does a lot to free us up to do more value added domain-specific work. Terraform has emerged as a key tool to manage AWS resources - so much so that it really adds a lot to the value prop of AWS itself. I can do stuff with Terraform that was only aspirational until it existed. Personally, I wouldn't plan on using on-prem except for niche applications that involve heavy data streams from local hardware. In the time that I've spent with companies working on AWS, I've seen a number of other companies waste lots of time and resources on heterogeneous strategies while complaining about their AWS bill - which was high because someone got so fed up with IT dysfunction, they went and used AWS but left behind inefficiently configured resources that were on all the time. Cloud often ends up being an escape hatch for teams that are not adequately served by dogmatic IT departments.
- dimitrios1 6y agoThe key glue to a lot of what you mentioned missing in your list to make a lot of this work for outside consumers: API Gateway, arguably their worst product.
- ak217 6y agoI agree, they really need to replace that product. It's not built to the standard that users expect from AWS. They have burned a lot of goodwill on that one. That said, it's only truly necessary for Lambda and while it's frustrating and painful there, it's usually not a complete showstopper.
- Aeolun 6y agoWhat is so terrible about API gateway? Aside from the price, but they kind of solved that with HTTP gateway.
- hinkley 6y agoGiven the situation with Amazon Essentials, anyone who sells a product made substantially of stringing together AWS technology may find themselves cut out of the picture soon enough. Many other companies have ended up competing with their own vendors. The only thing unique about Amazon is that they are boiling the frog much more slowly than previous generations of companies have.