10 ms·

Serverless to monolith – Should serverless lovers be worried?

83 points by fagnerbrack 3y ago

arter4 3y ago
This is the main insight I got: >the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered. Lesson learned: ensure your design is sane before turning your PoC into a full blown production system.
- PathOfEclipse 3y ago
  The real lesson to be learned is to always assume your PoC will, if successful, get shipped to production pretty much as-is!
- miked85 3y ago
  PoCs turning into full blown production systems happens quite often, even if not planned.
berbec 3y ago
hugged. https://archive.is/EDyUx https://archive.is/EDyUx
cagenut 3y ago
you can run a giant multi-gb monolith serverlessly. just because you use lambda doesnt mean you need to use dozens of them.
- piyh 3y ago
  Agreed, serverless and monolith are not on the same spectrum
deleted 3y ago
[deleted]
Scubabear68 3y ago
It is almost ridiculously common anti-pattern these days to see teams stringing lambda functions together for no good reason other then “cuz serverless”, or “microservices”, for what really should be just function calls in a single process. Developers are losing touch on what process boundaries mean, and the result is brittle and poor performing systems.
- intothemild 3y ago
  Hype driven development.
- goostavos 3y ago
  I regularly review design from teams that turn reading a file from S3 and putting into into a database into, well, something that requires a design review. Some horrific monstrosity of step functions, lambdas, and glue. All because they've been told that serverless is the correct way to build "scalable" software. If every time you add a new feature to your application you also have to add new infrastructure, I feel very, very strongly that you're doing it wrong. Sometimes? Sure. Every time? Embarrassing. Cloud providers are surely pleased as punch that there is now an entire generation of developers that will argue vehemently that infrastructure that only grows over time is not only healthy, but the best way to build scalable software. I have no idea what to call it other than "Marketing Material Driven Development".
  - BXlnt2EachOther 3y ago
    I have heard "Promotion-Driven Development" (when it's from the team/eng side)
  - hawk_ 3y ago
    Resume driven development. And on the flipside - One of the funniest justifications I have heard for companies using new fangled untried tech is to attract talent - no one wants to work on boring tech apparently for the fear of being outdated.
    - 3y ago
angarg12 3y ago
That article has been analyzed to death. Here is a good one from another ex-Amazon. https://news.ycombinator.com/item?id=35853148 https://news.ycombinator.com/item?id=35853148
SoftTalker 3y ago
Serverless always seemed to me like a stupid name. Of course there's a server. It's just not your server.
- atorodius 3y ago
  oh wow, I never knew I was always vaguely irritated by the term and never understood what it meant bc I was like „there must be some server??“ (not working in web dev)
  - SrslyJosh 3y ago
    Yep, you're essentially writing a module that's going to be imported by a server someone else wrote. The server will call the appropriate functions in your module based on the requests it receives, and Amazon gets a bigger check at the end of the month. I think there are probably reasonable use cases for this, like automation that's going to run infrequently, but it seems like a terrible way to implement a typical application that's going to receive frequent requests.
    - ElFitz 3y ago
      > and Amazon gets a bigger check at the end of the month. Or no check at all since it scales to zero and the free tier is ridiculously high for infrequent / low usage.
      - EdwardDiego 3y ago
        Which is custom designed for extracting money from corporations. PoC works, costs nothing, code delivered much faster. CTO impressed. PoC tidied up and rolled out to production. CTO very not impressed when quarterly expenses are reviewed, but customers are relying on this now.
    - orwin 3y ago
      > I think there are probably reasonable use cases for this, like automation that's going to run infrequently Exactly our usecase. Then the network admins wanted to click button and select their template from lists rather than having to write json, so website it was, and now we run a MVC on lambda. It isn't used that much (at most, you'll get 20 simulaneous users, and its always during a short time period, most of the time nobody use it). Don't know if it's the best choice. Honestly it seems fine.
talideon 3y ago
Not unless they're service providers.
beabetterdev 3y ago
Daniel here, author of this article. Thanks for re-sharing. I was quite disappointed to see the negative community reaction to the original Amazon article. I felt that those knocking it didn't analyze the circumstances that lead to the Serverless Architecture's gaps and just saw some problems with it and assumed "serverless bad". In reality, the series of decisions made a lot of sense. Take something that exists, try using it, learn from it, and make better decisions as a result of it. It just so happened that in this case, serverless architecture was the wrong tool for the job and dedicated machines were a better fit. Anyways, happy to see my take being shared. If you enjoy this kind of content, I have a newsletter and Youtube channel where I discuss / make videos about more AWS & cloud concepts: Newsletter: https://mailinglist.beabetterdev.com https://mailinglist.beabetterdev.com Youtube: https://www.youtube.com/c/BeABetterDev https://www.youtube.com/c/BeABetterDev Cheers, Daniel
- commandersaki 3y ago
  Thanks for the video format of the post, I found it surprisingly refreshing.
- blq10 3y ago
  I think serverless could be the right fit, something like detectors seems ideal. Theyre somewhat isolated individually, etc etc. I kind of want to take a Crack at what Prime Video did except serverless from a green field perspective, and I'm willing to bet I wind up provisioning a RedPanda cluster or similar and having them feed detector lambdas. The bottleneck and cost here appeared to be storing intermediate results and the orchestration. Orchestration is solved by using something like SQS or what have you. Intermediate results are somewhat harder depending on size - but I think you could get there and it'd be an interesting exercise.
- throwaway2990 3y ago
  [flagged]
  - adamckay 3y ago
    But isn't this an example of even AWS developers using cloud services wrong? They used a serverless approach (which should have excellent scalability -- it's one of the most common marketing lines of it!) then had to switch architecture because they realised it wasn't scalable.
phendrenad2 3y ago
I was downvoted this morning for suggestion that developers need to focus on objectively measurable results and stop insisting on using their favorite language or framework or paradigm. I guess things are much worse than I thought, because the headline here simply assumes as a given that "serverless lovers" are a thing that exists and are worth taking seriously.
Havoc 3y ago
Right tool for the job as always...sometimes that's serverless, sometimes that's monoliths.
Gaita 3y ago
When it comes to complex flows with many branches serverless really shines. It makes it easier to manage such complexity. For simpler flows serverless might just be an overkill. In short, there is a place for both.
- lacasito25 3y ago
  why is having those complex flows done in a graphical UI better than just in code, where you can easily version them / test them?
  - blq10 3y ago
    It makes it a lot clearer where logs, alerts, etc are coming from and it makes them easier to test - because the process flow of a given instruction is in the architecture. Verifying that the call patterns between the functions are correct is challenging sometimes - but it helps there to have the architecture mapped out visually.
- anonzzzies 3y ago
  There is no reason you cannot do this with a monolith. We and many others do every day.
lysergic 3y ago
During my 3 years as a backend developer at my current job I've seen a lot of application of "serverless" stuff from AWS, and I've also used quite a lot of those services myself. Personally I would never use something like Lambda for compute, unless it's something that needs to integrate between another AWS service and something else (likely also a service.) Also it seems like the "serverless" hype started (atleast at my employer), because some higher ups didn't trust the developers in my department, when they first had an AWS account provisioned. My general feeling about picking the right tool to solve a problem is to, firstly, pick the problem apart. Then you can evaluate technologies that can solve your problem. This includes code that you already own and it should most likely be this one that you pick. After that you can start to look at other technologies to bring in. Arguably it's nice that with AWS you can spin something up relatively quickly, however sometimes I'm also stumped by their limitations and what workarounds that seems to pop up as a result of that.
- blq10 3y ago
  The big place it shines for compute tasks is workloads with high variability. As long as you can select a p99 for timeouts that's sane, lambda does a really good job handling highly heterogeneous workflows in general. So if a request could take 10ms or multiple minutes depending on Reasons(TM) - lambda makes a lot of sense because you don't have to overprovision for theoretical peak demand, and a swamp of expensive requests is isolated from your cheap requests.
wodenokoto 3y ago
This articles refers to “100s of channels” and “100+ streams”. I cant figure out what that refers to. From a user perspective there is only 1 channel (maybe one per region, but that still doesn’t hit 100s) And for streams I can’t think of any kind of definition of stream that wouldn’t have orders of magnitude larger number associated with it on prime.
EdwardDiego 3y ago
Yes, serverless lovers should be worried. They should have always been worried. About just how much money their ideas have thrown at $CLOUD that could've been used for things other than helping build Bezos' Freudian Me-Too Ego Trip What Go Zoom. Or how much fragility and painfully slow development cycles have been inflicted on people in their org by the inevitable tight coupling to $SPECIFIC_PRODUCT_IN_SPECIFIC_REGION_OF_SPECIFIC_PROVIDER. And, they should be worried that they've drunk so deeply of the Kool Aid provided at $VENDOR_CONFERENCE_PROBABLY_IN_VEGAS that they considered the cost and the friction a worthwhile trade-off.
hk1337 3y ago
Severless is not going anywhere but hopefully the thoughts to go all-in with serverless on a new project dies.
andrewstuart 3y ago
Premature microservices/serverless architecture is the root of all evil. A fast modern computer will give a whole lotta vertical scaling. Even more so if you use a compiled language. I wrote a web application with rust using actix that did about 7 million requests per second.
- rajamaka 3y ago
  You can write a web application in any language using any library that does 7m rps
- bobobob420 3y ago
  How many instances or containers to handle that out or curiosity? And how long does the average request take?
  - andrewstuart 3y ago
    No instances or containers, just running bare metal. I don't know how long each request took.
mtl_design 3y ago
I feel like the hype caught up a lot of folks off guard and swayed them to do something that doesn't necessarily make sense. For example, building a simple web app (single page app, or whatever) using the hype around microservices, serverless, nosql, etc. is bound to fail. Fortunately, I see a trend in more "advanced" companies to stay away from microservices/lambdas in the first iterations of their products and embrace good old modular monolith with a simple architecture.
skyzyx 3y ago
Idiots. Idiots everywhere. Serverless is great for event handling and bursty workloads. It is not efficient for a long running or constantly running processes. That was never what it was designed for. There is a time and a place for serverless.
joaonmatos 3y ago
ECS on Fargate is still serverless. It seems that people think only Lambda is serverless, when serverless just means "product category where you pay for the usage directly, instead of for the resources provisioned to serve it". Lambda is a burstable, short-lived compute service; it's only natural that it's not well suited to situations where you have predictably sustained high throughput and homogeneous task requirements. To not even talk about how dumb the original architecture was for shuffling frames in and out of S3.

83 points by fagnerbrack 3y ago

This is the main insight I got: >the original architecture was based on a tool that was not designed for scale. Based on the article, it read as if this tool was used for diagnostics to perform ad-hoc assessments of stream quality. This means it likely wasn’t designed for scale or put through the pressure tests of a formal design document / review. If that were to happen, any run of the mill Software Engineer would have been able to recognize the obvious scaling / cost bottlenecks that would be encountered. Lesson learned: ensure your design is sane before turning your PoC into a full blown production system.

PathOfEclipse 3y ago
The real lesson to be learned is to always assume your PoC will, if successful, get shipped to production pretty much as-is!
miked85 3y ago
PoCs turning into full blown production systems happens quite often, even if not planned.

berbec 3y ago

hugged. https://archive.is/EDyUx https://archive.is/EDyUx

cagenut 3y ago

you can run a giant multi-gb monolith serverlessly. just because you use lambda doesnt mean you need to use dozens of them.

piyh 3y ago
Agreed, serverless and monolith are not on the same spectrum

deleted 3y ago

[deleted]

Scubabear68 3y ago

It is almost ridiculously common anti-pattern these days to see teams stringing lambda functions together for no good reason other then “cuz serverless”, or “microservices”, for what really should be just function calls in a single process. Developers are losing touch on what process boundaries mean, and the result is brittle and poor performing systems.

intothemild 3y ago
Hype driven development.
goostavos 3y ago
I regularly review design from teams that turn reading a file from S3 and putting into into a database into, well, something that requires a design review. Some horrific monstrosity of step functions, lambdas, and glue. All because they've been told that serverless is the correct way to build "scalable" software. If every time you add a new feature to your application you also have to add new infrastructure, I feel very, very strongly that you're doing it wrong. Sometimes? Sure. Every time? Embarrassing. Cloud providers are surely pleased as punch that there is now an entire generation of developers that will argue vehemently that infrastructure that only grows over time is not only healthy, but the best way to build scalable software. I have no idea what to call it other than "Marketing Material Driven Development".
- BXlnt2EachOther 3y ago
  I have heard "Promotion-Driven Development" (when it's from the team/eng side)
- hawk_ 3y ago
  Resume driven development. And on the flipside - One of the funniest justifications I have heard for companies using new fangled untried tech is to attract talent - no one wants to work on boring tech apparently for the fear of being outdated.
  - 3y ago

angarg12 3y ago

That article has been analyzed to death. Here is a good one from another ex-Amazon. https://news.ycombinator.com/item?id=35853148 https://news.ycombinator.com/item?id=35853148

SoftTalker 3y ago

Serverless always seemed to me like a stupid name. Of course there's a server. It's just not your server.

atorodius 3y ago
oh wow, I never knew I was always vaguely irritated by the term and never understood what it meant bc I was like „there must be some server??“ (not working in web dev)
- SrslyJosh 3y ago
  Yep, you're essentially writing a module that's going to be imported by a server someone else wrote. The server will call the appropriate functions in your module based on the requests it receives, and Amazon gets a bigger check at the end of the month. I think there are probably reasonable use cases for this, like automation that's going to run infrequently, but it seems like a terrible way to implement a typical application that's going to receive frequent requests.
  - ElFitz 3y ago
    > and Amazon gets a bigger check at the end of the month. Or no check at all since it scales to zero and the free tier is ridiculously high for infrequent / low usage.
    - EdwardDiego 3y ago
      Which is custom designed for extracting money from corporations. PoC works, costs nothing, code delivered much faster. CTO impressed. PoC tidied up and rolled out to production. CTO very not impressed when quarterly expenses are reviewed, but customers are relying on this now.
  - orwin 3y ago
    > I think there are probably reasonable use cases for this, like automation that's going to run infrequently Exactly our usecase. Then the network admins wanted to click button and select their template from lists rather than having to write json, so website it was, and now we run a MVC on lambda. It isn't used that much (at most, you'll get 20 simulaneous users, and its always during a short time period, most of the time nobody use it). Don't know if it's the best choice. Honestly it seems fine.

talideon 3y ago

Not unless they're service providers.

beabetterdev 3y ago

Daniel here, author of this article. Thanks for re-sharing. I was quite disappointed to see the negative community reaction to the original Amazon article. I felt that those knocking it didn't analyze the circumstances that lead to the Serverless Architecture's gaps and just saw some problems with it and assumed "serverless bad". In reality, the series of decisions made a lot of sense. Take something that exists, try using it, learn from it, and make better decisions as a result of it. It just so happened that in this case, serverless architecture was the wrong tool for the job and dedicated machines were a better fit. Anyways, happy to see my take being shared. If you enjoy this kind of content, I have a newsletter and Youtube channel where I discuss / make videos about more AWS & cloud concepts: Newsletter: https://mailinglist.beabetterdev.com https://mailinglist.beabetterdev.com Youtube: https://www.youtube.com/c/BeABetterDev https://www.youtube.com/c/BeABetterDev Cheers, Daniel

commandersaki 3y ago
Thanks for the video format of the post, I found it surprisingly refreshing.
blq10 3y ago
I think serverless could be the right fit, something like detectors seems ideal. Theyre somewhat isolated individually, etc etc. I kind of want to take a Crack at what Prime Video did except serverless from a green field perspective, and I'm willing to bet I wind up provisioning a RedPanda cluster or similar and having them feed detector lambdas. The bottleneck and cost here appeared to be storing intermediate results and the orchestration. Orchestration is solved by using something like SQS or what have you. Intermediate results are somewhat harder depending on size - but I think you could get there and it'd be an interesting exercise.
throwaway2990 3y ago
[flagged]
- adamckay 3y ago
  But isn't this an example of even AWS developers using cloud services wrong? They used a serverless approach (which should have excellent scalability -- it's one of the most common marketing lines of it!) then had to switch architecture because they realised it wasn't scalable.

phendrenad2 3y ago

I was downvoted this morning for suggestion that developers need to focus on objectively measurable results and stop insisting on using their favorite language or framework or paradigm. I guess things are much worse than I thought, because the headline here simply assumes as a given that "serverless lovers" are a thing that exists and are worth taking seriously.

Havoc 3y ago

Right tool for the job as always...sometimes that's serverless, sometimes that's monoliths.

Gaita 3y ago

When it comes to complex flows with many branches serverless really shines. It makes it easier to manage such complexity. For simpler flows serverless might just be an overkill. In short, there is a place for both.

lacasito25 3y ago
why is having those complex flows done in a graphical UI better than just in code, where you can easily version them / test them?
- blq10 3y ago
  It makes it a lot clearer where logs, alerts, etc are coming from and it makes them easier to test - because the process flow of a given instruction is in the architecture. Verifying that the call patterns between the functions are correct is challenging sometimes - but it helps there to have the architecture mapped out visually.
anonzzzies 3y ago
There is no reason you cannot do this with a monolith. We and many others do every day.

lysergic 3y ago

During my 3 years as a backend developer at my current job I've seen a lot of application of "serverless" stuff from AWS, and I've also used quite a lot of those services myself. Personally I would never use something like Lambda for compute, unless it's something that needs to integrate between another AWS service and something else (likely also a service.) Also it seems like the "serverless" hype started (atleast at my employer), because some higher ups didn't trust the developers in my department, when they first had an AWS account provisioned. My general feeling about picking the right tool to solve a problem is to, firstly, pick the problem apart. Then you can evaluate technologies that can solve your problem. This includes code that you already own and it should most likely be this one that you pick. After that you can start to look at other technologies to bring in. Arguably it's nice that with AWS you can spin something up relatively quickly, however sometimes I'm also stumped by their limitations and what workarounds that seems to pop up as a result of that.

blq10 3y ago
The big place it shines for compute tasks is workloads with high variability. As long as you can select a p99 for timeouts that's sane, lambda does a really good job handling highly heterogeneous workflows in general. So if a request could take 10ms or multiple minutes depending on Reasons(TM) - lambda makes a lot of sense because you don't have to overprovision for theoretical peak demand, and a swamp of expensive requests is isolated from your cheap requests.

wodenokoto 3y ago

This articles refers to “100s of channels” and “100+ streams”. I cant figure out what that refers to. From a user perspective there is only 1 channel (maybe one per region, but that still doesn’t hit 100s) And for streams I can’t think of any kind of definition of stream that wouldn’t have orders of magnitude larger number associated with it on prime.

EdwardDiego 3y ago

Yes, serverless lovers should be worried. They should have always been worried. About just how much money their ideas have thrown at $CLOUD that could've been used for things other than helping build Bezos' Freudian Me-Too Ego Trip What Go Zoom. Or how much fragility and painfully slow development cycles have been inflicted on people in their org by the inevitable tight coupling to $SPECIFIC_PRODUCT_IN_SPECIFIC_REGION_OF_SPECIFIC_PROVIDER. And, they should be worried that they've drunk so deeply of the Kool Aid provided at $VENDOR_CONFERENCE_PROBABLY_IN_VEGAS that they considered the cost and the friction a worthwhile trade-off.

hk1337 3y ago

Severless is not going anywhere but hopefully the thoughts to go all-in with serverless on a new project dies.

andrewstuart 3y ago

Premature microservices/serverless architecture is the root of all evil. A fast modern computer will give a whole lotta vertical scaling. Even more so if you use a compiled language. I wrote a web application with rust using actix that did about 7 million requests per second.

rajamaka 3y ago
You can write a web application in any language using any library that does 7m rps
bobobob420 3y ago
How many instances or containers to handle that out or curiosity? And how long does the average request take?
- andrewstuart 3y ago
  No instances or containers, just running bare metal. I don't know how long each request took.

mtl_design 3y ago

I feel like the hype caught up a lot of folks off guard and swayed them to do something that doesn't necessarily make sense. For example, building a simple web app (single page app, or whatever) using the hype around microservices, serverless, nosql, etc. is bound to fail. Fortunately, I see a trend in more "advanced" companies to stay away from microservices/lambdas in the first iterations of their products and embrace good old modular monolith with a simple architecture.

skyzyx 3y ago

Idiots. Idiots everywhere. Serverless is great for event handling and bursty workloads. It is not efficient for a long running or constantly running processes. That was never what it was designed for. There is a time and a place for serverless.

joaonmatos 3y ago

ECS on Fargate is still serverless. It seems that people think only Lambda is serverless, when serverless just means "product category where you pay for the usage directly, instead of for the resources provisioned to serve it". Lambda is a burstable, short-lived compute service; it's only natural that it's not well suited to situations where you have predictably sustained high throughput and homogeneous task requirements. To not even talk about how dumb the original architecture was for shuffling frames in and out of S3.