4 ms·

Over the years I’ve spent a lot of time talking engineers and managers out of using serverless AWS options for various reasons. I’ve found that most non-infra

by scrose 2y ago

Over the years I’ve spent a lot of time talking engineers and managers out of using serverless AWS options for various reasons. I’ve found that most non-infra focused engineers and managers see serverless marketed as “simpler” and “cheaper”. It’s often the opposite, but most people don’t see that until after they’ve built their infrastructure around that, get locked in, and then start seeing the surprise bills, difficult to diagnose system failures, and hard-limitations start rolling in. A bit of early skepticism, and alternative solutions with a long-term perspective in mind, often go a long way.

xyzzy123 2y ago
It's like, the complexity has to live somewhere. I've seen successful serverless designs but the complexity gets pushed out of code and into service configuration / integration (it becomes arch spaghetti). These systems are difficult to properly test until deployed to cloud. Also, yeah, total vendor lock in. It works for some teams but is not my preference.
irjustin 2y ago
If you have a low traffic or % of server ultiziation such as B2B applications. "Full Container on Serverless" can be insanely cheap. Such as FastAPI, Django, Rails etc running all of these on Lambda when you've only got a few hits during the day and almost none at night, is very cost effective.
- jhot 2y ago
  We do this at current job for most of our internal tools. Not a setup I would choose on my own, but serviceable. Using a simple handler function that uses mangum [0] to translate the event into a request compatible with FastAPI, it mostly Just Works TM the same in AWS as it does locally. The trade off comes with a bit harder troubleshooting and there are some cases where it can be difficult to reproduce a bug locally because of the different server architectures. 0: https://github.com/Kludex/mangum https://github.com/Kludex/mangum
- bigfatkitten 2y ago
  It can be, but a whole lot of people fail to do this analysis properly to see whether it will be for their use case, and they get burned.
alias_neo 2y ago
As with many of these things, I've seen it time and time again; the initial setup can be simple, perhaps that's a good thing for an org that needs to get moving quickly, but very soon you start to see the limitations and the more advanced stuff just isn't possible and now you're stuck, locked in so have to rebuild from the ground up. Sometimes you have to do the (over) engineering part yourself, so that someone else isn't making the decisions on your behalf.
liendolucas 2y ago
What it's also surprising is people getting excited and "certified" on AWS (and attending AWS conferences, lol), job postings requiring you to "know" AWS for a developer position. Why on earth do I have to know AWS to develop software? Isn't it that supposed to be covered by DevOps or sysadmins? If one word could define AWS that would be: overengineer. The industry definitely does not need all that machinery to build things, probably a fraction of what is offered and way way simpler.
- tietjens 2y ago
  Because the roles are increasingly blurring, and require both the dev and the ops knowledge. AWS gives you a lot of power if you buy into it, but of course that comes with a whole set of tradeoffs. There won’t be less cloud in the future, no matter personal feelings about it.
- larusso 2y ago
  Because if you hire a DevOps after the original meaning then he needs to know AWS (assuming that’s the cloud vendor the company posting is using) DevOps means develop and operate. That was the raging new concept. Since actual sysadmin work of setting up hardware is no longer needed when hosting on AWS. So the developer takes the part of hosting and operation. But now that cloud infrastructure became so damn complicated and all, most DevOps define the dev as developing and maintaining the Zoo of tools and configurations. No time for actual development of the product. This is handled by another team. And we are back full circle to the times before DevOps. Our company still runs the old style of the definition and it is manageable. [Edit typo]
moltar 2y ago
Just a counterpoint. But my experience has been the opposite. Taking a legacy application that runs on a single server and is struggling to scale with spaghetti of config files all over the box and security issues that have gone unpatched because no one wants to touch the box as it’s too dangerous to change anything, as everything is intertwined. No backups or unverified backups. No redundancy. No reproducible environments so things work on local but not on remote. Takes devs days to pin the issues. Then break it down and replace with individual fully managed AWS services that can scale independently and I don’t need to worry about upgrades. Yeah, the total bill for services is usually higher. But that’s because now there’s actually someone (AWS) maintaining each service, rather than completely neglecting it. They key to all this is to use IaC such as AWS CDK to have code be the single source of truth. No ClickOps!
- scrose 2y ago
  I mostly understand what you’re saying and I hope I didn’t come across as saying serverless is *never* a good idea. In my experience it’s just very often not the best choice. I will always keep an open mind about solutions, but serverless often comes with many shortcomings that teams don’t notice until it’s too late. I’ve seen it happen enough times that I feel I must keep a healthy dose of skepticism when anyone proposes changing a small part of stack for serverless options. I’ve been working with AWS for about 8 years now and have worked on supporting dozens of projects across 5 different orgs ranging from small 5-person shops to 1000+ people. I have only seen a handful of cases where serverless makes sense. > Taking a legacy application that runs on a single server and is struggling to scale with spaghetti of config files all over the box and security issues that have gone unpatched because no one wants to touch the box as it’s too dangerous to change anything, as everything is intertwined. No backups or unverified backups. No redundancy. No reproducible environments so things work on local but not on remote. Takes devs days to pin the issues. All of these issues are easily possible with many serverless options. What it really sounds like is your org went from a legacy architecture that built up a ton of tech debt over the course of years and had no one owning the basics (ie backups) to a new one with less tech debt since it was built specifically around current issues you were facing. In 3-5 years I wouldn’t be surprised to hear the pendulum swinging back in the other direction as staffing and resources change hands and new requirements emerge.
kikimora 2y ago
My experience - system went down every time we have significant load. Reasons were various, but all triggered by load. Switched to ECS + Aurora - problem gone, bill has slightly increased.
qwertycrackers 2y ago
Our team kinda thinks the same thing about serverless but despite that we have some things built with it. And the paradoxical thing is that this issues have just never materialized, the serverless stuff is overwhelmingly the most stable part of our application. It's kinda weird and we don't fully trust it but empirically serverless works as advertised.
- franktankbank 2y ago
  How long have you been running on serverless?

by scrose 2y ago

xyzzy123 2y ago

It's like, the complexity has to live somewhere. I've seen successful serverless designs but the complexity gets pushed out of code and into service configuration / integration (it becomes arch spaghetti). These systems are difficult to properly test until deployed to cloud. Also, yeah, total vendor lock in. It works for some teams but is not my preference.

irjustin 2y ago

If you have a low traffic or % of server ultiziation such as B2B applications. "Full Container on Serverless" can be insanely cheap. Such as FastAPI, Django, Rails etc running all of these on Lambda when you've only got a few hits during the day and almost none at night, is very cost effective.

jhot 2y ago
We do this at current job for most of our internal tools. Not a setup I would choose on my own, but serviceable. Using a simple handler function that uses mangum [0] to translate the event into a request compatible with FastAPI, it mostly Just Works TM the same in AWS as it does locally. The trade off comes with a bit harder troubleshooting and there are some cases where it can be difficult to reproduce a bug locally because of the different server architectures. 0: https://github.com/Kludex/mangum https://github.com/Kludex/mangum
bigfatkitten 2y ago
It can be, but a whole lot of people fail to do this analysis properly to see whether it will be for their use case, and they get burned.

alias_neo 2y ago

As with many of these things, I've seen it time and time again; the initial setup can be simple, perhaps that's a good thing for an org that needs to get moving quickly, but very soon you start to see the limitations and the more advanced stuff just isn't possible and now you're stuck, locked in so have to rebuild from the ground up. Sometimes you have to do the (over) engineering part yourself, so that someone else isn't making the decisions on your behalf.

liendolucas 2y ago

What it's also surprising is people getting excited and "certified" on AWS (and attending AWS conferences, lol), job postings requiring you to "know" AWS for a developer position. Why on earth do I have to know AWS to develop software? Isn't it that supposed to be covered by DevOps or sysadmins? If one word could define AWS that would be: overengineer. The industry definitely does not need all that machinery to build things, probably a fraction of what is offered and way way simpler.

tietjens 2y ago
Because the roles are increasingly blurring, and require both the dev and the ops knowledge. AWS gives you a lot of power if you buy into it, but of course that comes with a whole set of tradeoffs. There won’t be less cloud in the future, no matter personal feelings about it.
larusso 2y ago
Because if you hire a DevOps after the original meaning then he needs to know AWS (assuming that’s the cloud vendor the company posting is using) DevOps means develop and operate. That was the raging new concept. Since actual sysadmin work of setting up hardware is no longer needed when hosting on AWS. So the developer takes the part of hosting and operation. But now that cloud infrastructure became so damn complicated and all, most DevOps define the dev as developing and maintaining the Zoo of tools and configurations. No time for actual development of the product. This is handled by another team. And we are back full circle to the times before DevOps. Our company still runs the old style of the definition and it is manageable. [Edit typo]

moltar 2y ago

Just a counterpoint. But my experience has been the opposite. Taking a legacy application that runs on a single server and is struggling to scale with spaghetti of config files all over the box and security issues that have gone unpatched because no one wants to touch the box as it’s too dangerous to change anything, as everything is intertwined. No backups or unverified backups. No redundancy. No reproducible environments so things work on local but not on remote. Takes devs days to pin the issues. Then break it down and replace with individual fully managed AWS services that can scale independently and I don’t need to worry about upgrades. Yeah, the total bill for services is usually higher. But that’s because now there’s actually someone (AWS) maintaining each service, rather than completely neglecting it. They key to all this is to use IaC such as AWS CDK to have code be the single source of truth. No ClickOps!

scrose 2y ago
I mostly understand what you’re saying and I hope I didn’t come across as saying serverless is *never* a good idea. In my experience it’s just very often not the best choice. I will always keep an open mind about solutions, but serverless often comes with many shortcomings that teams don’t notice until it’s too late. I’ve seen it happen enough times that I feel I must keep a healthy dose of skepticism when anyone proposes changing a small part of stack for serverless options. I’ve been working with AWS for about 8 years now and have worked on supporting dozens of projects across 5 different orgs ranging from small 5-person shops to 1000+ people. I have only seen a handful of cases where serverless makes sense. > Taking a legacy application that runs on a single server and is struggling to scale with spaghetti of config files all over the box and security issues that have gone unpatched because no one wants to touch the box as it’s too dangerous to change anything, as everything is intertwined. No backups or unverified backups. No redundancy. No reproducible environments so things work on local but not on remote. Takes devs days to pin the issues. All of these issues are easily possible with many serverless options. What it really sounds like is your org went from a legacy architecture that built up a ton of tech debt over the course of years and had no one owning the basics (ie backups) to a new one with less tech debt since it was built specifically around current issues you were facing. In 3-5 years I wouldn’t be surprised to hear the pendulum swinging back in the other direction as staffing and resources change hands and new requirements emerge.

kikimora 2y ago

My experience - system went down every time we have significant load. Reasons were various, but all triggered by load. Switched to ECS + Aurora - problem gone, bill has slightly increased.

qwertycrackers 2y ago

Our team kinda thinks the same thing about serverless but despite that we have some things built with it. And the paradoxical thing is that this issues have just never materialized, the serverless stuff is overwhelmingly the most stable part of our application. It's kinda weird and we don't fully trust it but empirically serverless works as advertised.

franktankbank 2y ago
How long have you been running on serverless?