6 ms·

I'll throw myself into the gauntlet here because its my job. (Chris Munns - Senior Developer Advocate for Serverless @ AWS)(munns@amazon.com) A lot of the comm

by munns 9y ago

I'll throw myself into the gauntlet here because its my job. (Chris Munns - Senior Developer Advocate for Serverless @ AWS)(munns@amazon.com) A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least. Fwiw I've run my own metal at some fairly large websites(can go find me on Linkedin) and I can honestly say that to discredit the amount of time you spend on solving the above capabilities WELL is to probably toss more than 120% of the cost out the window. People time to manage what goes on the hardware is almost always higher than the cost of the hardware. I've spent 5+ years working with everyone from startups up through massive enterprises and only the top 10% does any of this well when doing it themselves. Taking some of the other examples here, let's say you do try and solve this with 2 virtual/physical servers from a traditional hosting provider. What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls. Those last few all scale on their own too (oh you've never filled a harddrive with logs before?). Are you going to run all of your software on these 2-3 hosts? What happens when they go down? How quickly are you recovering? What impact is there to your customers during this time? What value do you place on that downtime? How many of your tools just failed? Lost data? How about patching the OS, the software you run on it, your tools, your languages/packages? In busier environments where I ran my own logging, monitoring, metrics, alarming, backups, software, firewalls, etc + the operating systems those things ran on + the modules/packages most developers use these days in languages such as node.js and python, you are talking about whole days in a week to track, test, and understand these changes and the risks associated. I'm a #serverless zealot by trade sure, and I am happy to be called out on that, but these comparisons of a very high level platform that essentially removes most of the day to day care and feeding of your own "iron" can not be compared to running your own iron at just that iron's cost.

hedora 9y ago
At $370/month, you're right: Operating all of these services is definitely costing more in human time than machine time. I think the confusion stems from the fact that we don't really know how expensive a "request" is. Many applications can do 1000's of requests per second per machine (say 15K/sec). Extrapolating the 15 requests per second = $370/month in the article, my hypothetical one machine application will cost $370K/month on lambda. For that, I can rent colo space, hire people to run all the services you mentioned, and gold plate the racks while I'm at it. That is crazy overhead for one machine worth of workload. The GB*sec of a request in my extrapolated workload is certainly wrong, but I suspect a lot of people ran this calculation in their heads. I don't really know what a request does in this context, except that version N-1 was 100x more expensive than the current one. I think the right takeaway from the article is that spending some time to optimize workloads that are costing O(one engineer salary) to operate is often worth the effort, and, for low traffic stuff, it is hard to beat one-size-fits-all infrastructure like lambda.
- munns 9y ago
  It's true, "spring cleaning" from time to time can pay back in ways that people don't expect once they clean up months and years worth of additive work on top of something else. I agree that is a big take away. On per request pricing this is something you can do with Lambda now semi-trivially. You can figure out average Lambda execution cost based on duration of execution and memory you have configured + API-GW related request costs. Depending on your DB you can potentially tie this back as well (harder with some DBs vs. others) I'd say that while we see cost savings for most serverless platform customers, like an overwhelming amount honestly, we also see companies adopting it for the benefits of time to market, reduced operation burden, and the overall capabilities that they don't need to write themselves. Hard to ROI some of that. While this article did take on a cost based view I'd love to see more folks talk about how these "softer" benefits impact them. Thanks for your reply! - Chris
deleted 9y ago
[deleted]
kogepathic 9y ago
> A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least. I'm not here to pick a fight against AWS. You guys have built a powerful ecosystem and it's definitely applicable to certain business cases. Your points about building the same ecosystem as AWS on traditional hardware are absolutely correct, this is very difficult to get right. A lot of places simply don't. I have always said that AWS is great for loads where you need to start be able to easily scale up and down, something which is prohibitively expensive to do with your own servers. But what you might not see in your position inside Amazon, is that for most companies it's much easier to hire developers/sysadmins for "traditional systems" than it is to find people with deep AWS knowledge. Because AWS is such a large ecosystem the market of individuals with the domain specific AWS knowledge that your business needs is much smaller than regular old sysadmin/developers. This means either it's very hard to find the right people, or you have to pay them multiples of what you would pay a standard developer/sysadmin. It's the difficulty of finding people to fill the role that vexes me most. > What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls. Ah, but you are assuming people who are using AWS are adhering to best practices and have all these set up, tested, and working in production. I've worked at a few companies with on site and AWS presence, and I can say that none of them followed best practices all the time...
- munns 9y ago
  These are all true and valid points. True story: having AWS (or any cloud experience) on your resume today is a major boost to your money earning potential: A. because its rare to have several years experience. B. because the space is growing rapidly (again all of cloud). If you haven't started teaching yourself the basics of cloud computing platforms start today because your next job or the job after that WILL involve you running applications on public cloud. In my experience from pre-AWS days hiring good Ops people for systems, networking, etc is hard no matter what. I'd argue that the developer's role/day to day changes little in the move from traditional "on-prem" type infrastructure to cloud, even with serverless platforms in play. Generally speaking though I'd aim to hire people who are hungry to learn and have enough fundamental knowledge of networks, operating systems, and hardware to be able to ramp up on public cloud technologies. If you learned it one way, you can learn it this way and there's a massive amount of people here at AWS at least to help you along the way (training folks, support folks, solutions architects) as well as 3rd parties like ACloudGuru, Linux Academy, and others with training/consulting capabilities. The future of technology is moving towards public cloud and I'd argue serverless application development models whether you want to get on board or not. No one cried for the last blacksmith in the town when cars replaced horses. Best practices are hard for people to follow no matter the technology, and the easiest solution is rarely the "best" (see people leave SG's wide open, open S3 perms, etc). We at AWS know its our job to make security best practices easier to grok and implement and there are tools that help with that today (IAM testing/reviewing tools, trusted advisor as examples). There's more we can do here. Thanks for your reply! - Chris
athenot 9y ago
> I can honestly say that to discredit the amount of time you spend on solving the above capabilities WELL is to probably toss more than 120% of the cost out the window. Yes. Where I'm at, I like to believe we solve this reasonably well, and once you factor in all the redundancy for HA, DR & all the engineering around proper monitoring, security, log management... we find Lambda very attractive. Though in our case we tie Lambda back into our infrastructure, orgs that are expediently trying to solve a problem (greenfield projects) and don't have the staff to set all this stuff up, take a deep look at Lambda. At worse, you can always move your lambda functions into your own servers once you have the resources to devote to setting up all the stuff that goes for highly available setups.
sbov 9y ago
It's apples to apples depending upon the needs of the project and business. From my experience few projects can go 100% serverless. Once you give that up, much of what you talk about is back into your view anyways. Beyond that, cloud vs physical servers again depends upon the requirements of the project and business. We do a mix of cloud and physical servers, and I'm pushing our company to go more serverless (and cloud), but that could easily change based upon which projects gain traction. Some are high revenue like ecommerce, but others are more ad based and the cloud would probably dig into margins too much. We also aren't a VC backed startup nor are we a huge business so the needs are different. But at our scale, colocating, much less virtual servers, is not even a full time job. And when I say requirements, I mean actual requirements. Many like to talk about 100% uptime. Then most seem to build their project to only work on a single region of AWS. Single region SLA of AWS is 99.95%. Guess they didn't need that 100% uptime. Of course, multi region is much harder, which is why they do it. But still, I wish people would stop talking about benefits they aren't actually taking advantage of. It makes it difficult to separate the signal from the bullshit.
hartator 9y ago
The issue is $370 per month is still a lot of money. While you can probably archieve the same performance on $5-10 vm or bare metal server.
- merb 9y ago
  well it depends on what you want. High Availability comes with a cost. well it's good that more and more vm providers like DO allow you to create your own network. But many still do, so you need some kind of vpn to securly create a network on top.
  - hartator 9y ago
    I think cost do matter. Look at Wassap. They had only one bare metal server in production until they got acquired. The "Cloud" is sometimes deconnected from the reality.
  - tjholowaychuk 9y ago
    It's not highly available, in fact Lambda just went down in two regions.
Veratyr 9y ago
As someone who made one of the comparisons you're commenting on... > A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least. Yes, I acknowledge that buying a pair of dedicated servers will not give me the things you mention however these things are coming at a cost and while they're nice to have, when you're running at a scale where a mere 2 baremetal servers can handle peak load, I'm dubious that you really need these things enough to pay ~2x to have them. > What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls. HA: OVH has load balancers too. Logging: Log to disk, rotate logs, compress them and send them to somewhere like S3 for further analysis with a cron job if desired. Monitoring + Alerting: You have competitors like NewRelic and DataDog, they're pretty trivial to set up. Alternatives can be self hosted when things get too expensive. Deployment: Depends on the app. Auto-updates on the servers and a private package repository could be enough. Security Controls: Depends entirely on the application. Though not everyone needs all these things and AWS adds $200/month in additional expense relative to baremetal, which is a lot of savings to be had. > Are you going to run all of your software on these 2-3 hosts? What happens when they go down? How quickly are you recovering? Are you going to run all your software on AWS? What happens when AWS goes down? How quickly is AWS recovering? You have to deal with these problems either way, AWS is no silver bullet and you're going to need to have disaster recovery plans either way. Most dedicated hosting providers I'm aware of will give you a basic SLA to start with and a better one can be purchased later. That covers hardware issues. Software issues are up to you but from the sound of things, the author's software is little more than a proxy. > How about patching the OS, the software you run on it, your tools, your languages/packages? Turn on unattended upgrades, pay a sysadmin a couple hundred bucks once every couple years to handle major upgrades if it's too scary to do yourself. I'm under no illusions that paying AWS is always the wrong idea but the benefits of AWS come at high cost and not everyone is willing to pay it. I'm certainly not. I pay $60/mo for a colo (let's say $100/mo factoring in hardware) and to get the same capabilities on AWS, I'd be paying $23k for bandwidth, $1.4k for disk and $152 for CPU/RAM (an m3.2xlarge on a 3-year reserved plan), for a total of ~$24500/mo, or 245x what I'm paying now and more than enough to pay 2 full time engineers if I so wished.
tjholowaychuk 9y ago
Just my opinion, but I think API Gateway is massively over priced, considering a t2.nano could easily do the same job, and just slap that in a auto-scaling group to keep it running. API Gateway doesn't even come close to EC2. I could see maybe 200%, but not like 5000% increase. 39M requests per month is basically nothing. I'm just guessing, but maybe it's related to the inherent flaw that Lambda only processes with a concurrency of 1 (per func container), instead of utilizing Node / Golang concurrency capabilities.

by munns 9y ago

hedora 9y ago

At $370/month, you're right: Operating all of these services is definitely costing more in human time than machine time. I think the confusion stems from the fact that we don't really know how expensive a "request" is. Many applications can do 1000's of requests per second per machine (say 15K/sec). Extrapolating the 15 requests per second = $370/month in the article, my hypothetical one machine application will cost $370K/month on lambda. For that, I can rent colo space, hire people to run all the services you mentioned, and gold plate the racks while I'm at it. That is crazy overhead for one machine worth of workload. The GB*sec of a request in my extrapolated workload is certainly wrong, but I suspect a lot of people ran this calculation in their heads. I don't really know what a request does in this context, except that version N-1 was 100x more expensive than the current one. I think the right takeaway from the article is that spending some time to optimize workloads that are costing O(one engineer salary) to operate is often worth the effort, and, for low traffic stuff, it is hard to beat one-size-fits-all infrastructure like lambda.

munns 9y ago
It's true, "spring cleaning" from time to time can pay back in ways that people don't expect once they clean up months and years worth of additive work on top of something else. I agree that is a big take away. On per request pricing this is something you can do with Lambda now semi-trivially. You can figure out average Lambda execution cost based on duration of execution and memory you have configured + API-GW related request costs. Depending on your DB you can potentially tie this back as well (harder with some DBs vs. others) I'd say that while we see cost savings for most serverless platform customers, like an overwhelming amount honestly, we also see companies adopting it for the benefits of time to market, reduced operation burden, and the overall capabilities that they don't need to write themselves. Hard to ROI some of that. While this article did take on a cost based view I'd love to see more folks talk about how these "softer" benefits impact them. Thanks for your reply! - Chris

deleted 9y ago

[deleted]

kogepathic 9y ago

> A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least. I'm not here to pick a fight against AWS. You guys have built a powerful ecosystem and it's definitely applicable to certain business cases. Your points about building the same ecosystem as AWS on traditional hardware are absolutely correct, this is very difficult to get right. A lot of places simply don't. I have always said that AWS is great for loads where you need to start be able to easily scale up and down, something which is prohibitively expensive to do with your own servers. But what you might not see in your position inside Amazon, is that for most companies it's much easier to hire developers/sysadmins for "traditional systems" than it is to find people with deep AWS knowledge. Because AWS is such a large ecosystem the market of individuals with the domain specific AWS knowledge that your business needs is much smaller than regular old sysadmin/developers. This means either it's very hard to find the right people, or you have to pay them multiples of what you would pay a standard developer/sysadmin. It's the difficulty of finding people to fill the role that vexes me most. > What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls. Ah, but you are assuming people who are using AWS are adhering to best practices and have all these set up, tested, and working in production. I've worked at a few companies with on site and AWS presence, and I can say that none of them followed best practices all the time...

munns 9y ago
These are all true and valid points. True story: having AWS (or any cloud experience) on your resume today is a major boost to your money earning potential: A. because its rare to have several years experience. B. because the space is growing rapidly (again all of cloud). If you haven't started teaching yourself the basics of cloud computing platforms start today because your next job or the job after that WILL involve you running applications on public cloud. In my experience from pre-AWS days hiring good Ops people for systems, networking, etc is hard no matter what. I'd argue that the developer's role/day to day changes little in the move from traditional "on-prem" type infrastructure to cloud, even with serverless platforms in play. Generally speaking though I'd aim to hire people who are hungry to learn and have enough fundamental knowledge of networks, operating systems, and hardware to be able to ramp up on public cloud technologies. If you learned it one way, you can learn it this way and there's a massive amount of people here at AWS at least to help you along the way (training folks, support folks, solutions architects) as well as 3rd parties like ACloudGuru, Linux Academy, and others with training/consulting capabilities. The future of technology is moving towards public cloud and I'd argue serverless application development models whether you want to get on board or not. No one cried for the last blacksmith in the town when cars replaced horses. Best practices are hard for people to follow no matter the technology, and the easiest solution is rarely the "best" (see people leave SG's wide open, open S3 perms, etc). We at AWS know its our job to make security best practices easier to grok and implement and there are tools that help with that today (IAM testing/reviewing tools, trusted advisor as examples). There's more we can do here. Thanks for your reply! - Chris

athenot 9y ago

> I can honestly say that to discredit the amount of time you spend on solving the above capabilities WELL is to probably toss more than 120% of the cost out the window. Yes. Where I'm at, I like to believe we solve this reasonably well, and once you factor in all the redundancy for HA, DR & all the engineering around proper monitoring, security, log management... we find Lambda very attractive. Though in our case we tie Lambda back into our infrastructure, orgs that are expediently trying to solve a problem (greenfield projects) and don't have the staff to set all this stuff up, take a deep look at Lambda. At worse, you can always move your lambda functions into your own servers once you have the resources to devote to setting up all the stuff that goes for highly available setups.

sbov 9y ago

It's apples to apples depending upon the needs of the project and business. From my experience few projects can go 100% serverless. Once you give that up, much of what you talk about is back into your view anyways. Beyond that, cloud vs physical servers again depends upon the requirements of the project and business. We do a mix of cloud and physical servers, and I'm pushing our company to go more serverless (and cloud), but that could easily change based upon which projects gain traction. Some are high revenue like ecommerce, but others are more ad based and the cloud would probably dig into margins too much. We also aren't a VC backed startup nor are we a huge business so the needs are different. But at our scale, colocating, much less virtual servers, is not even a full time job. And when I say requirements, I mean actual requirements. Many like to talk about 100% uptime. Then most seem to build their project to only work on a single region of AWS. Single region SLA of AWS is 99.95%. Guess they didn't need that 100% uptime. Of course, multi region is much harder, which is why they do it. But still, I wish people would stop talking about benefits they aren't actually taking advantage of. It makes it difficult to separate the signal from the bullshit.

hartator 9y ago

The issue is $370 per month is still a lot of money. While you can probably archieve the same performance on $5-10 vm or bare metal server.

merb 9y ago
well it depends on what you want. High Availability comes with a cost. well it's good that more and more vm providers like DO allow you to create your own network. But many still do, so you need some kind of vpn to securly create a network on top.
- hartator 9y ago
  I think cost do matter. Look at Wassap. They had only one bare metal server in production until they got acquired. The "Cloud" is sometimes deconnected from the reality.
- tjholowaychuk 9y ago
  It's not highly available, in fact Lambda just went down in two regions.

Veratyr 9y ago

As someone who made one of the comparisons you're commenting on... > A lot of the comments here are comparing bare metal/virtual servers to a combination of two+ products with literally dozens of built in capabilities, security controls, built in scaling, monitoring, logging, deployment mechanisms, and built in high availability. These are not apples to apples comparisons in the least. Yes, I acknowledge that buying a pair of dedicated servers will not give me the things you mention however these things are coming at a cost and while they're nice to have, when you're running at a scale where a mere 2 baremetal servers can handle peak load, I'm dubious that you really need these things enough to pay ~2x to have them. > What now do you do for: high availability, nearly instant scalability, monitoring, logging, alarming, deployment and security controls. HA: OVH has load balancers too. Logging: Log to disk, rotate logs, compress them and send them to somewhere like S3 for further analysis with a cron job if desired. Monitoring + Alerting: You have competitors like NewRelic and DataDog, they're pretty trivial to set up. Alternatives can be self hosted when things get too expensive. Deployment: Depends on the app. Auto-updates on the servers and a private package repository could be enough. Security Controls: Depends entirely on the application. Though not everyone needs all these things and AWS adds $200/month in additional expense relative to baremetal, which is a lot of savings to be had. > Are you going to run all of your software on these 2-3 hosts? What happens when they go down? How quickly are you recovering? Are you going to run all your software on AWS? What happens when AWS goes down? How quickly is AWS recovering? You have to deal with these problems either way, AWS is no silver bullet and you're going to need to have disaster recovery plans either way. Most dedicated hosting providers I'm aware of will give you a basic SLA to start with and a better one can be purchased later. That covers hardware issues. Software issues are up to you but from the sound of things, the author's software is little more than a proxy. > How about patching the OS, the software you run on it, your tools, your languages/packages? Turn on unattended upgrades, pay a sysadmin a couple hundred bucks once every couple years to handle major upgrades if it's too scary to do yourself. I'm under no illusions that paying AWS is always the wrong idea but the benefits of AWS come at high cost and not everyone is willing to pay it. I'm certainly not. I pay $60/mo for a colo (let's say $100/mo factoring in hardware) and to get the same capabilities on AWS, I'd be paying $23k for bandwidth, $1.4k for disk and $152 for CPU/RAM (an m3.2xlarge on a 3-year reserved plan), for a total of ~$24500/mo, or 245x what I'm paying now and more than enough to pay 2 full time engineers if I so wished.

tjholowaychuk 9y ago

Just my opinion, but I think API Gateway is massively over priced, considering a t2.nano could easily do the same job, and just slap that in a auto-scaling group to keep it running. API Gateway doesn't even come close to EC2. I could see maybe 200%, but not like 5000% increase. 39M requests per month is basically nothing. I'm just guessing, but maybe it's related to the inherent flaw that Lambda only processes with a concurrency of 1 (per func container), instead of utilizing Node / Golang concurrency capabilities.