5 ms·

Having dealt with the pain and complexity of a 100+ lambda function app for the last 4 years, I must say this post definitely hits the spot wrt. the downsides o

Having dealt with the pain and complexity of a 100+ lambda function app for the last 4 years, I must say this post definitely hits the spot wrt. the downsides of FaaS serverless architectures. When starting out, these downsides are not really that visible. On the contrary, there is a very clear upside, which is that everything is free when you have low usage, and you have little to no maintenance. It is only later, when you have built a hot mess of lambda workflows, which become more and more rigid due to interdependencies, that you wish you had just gone the monolith route and spent the few extra hundreds on something self-managed. (Or even less now, e.g. on fly.io) A question for author: what if not using Elixir?

chrismccord 3y ago
I talk about FLAME outside elixir in one one of the sections in the blog. The tldr; is it's a generally applicable pattern for languages with a reasonable concurrency model. You likely won't get all the ergonomics that we get for free like functions with captured variable serialization, but you can probably get 90% of the way there in something like js, where you can move your modular execution to a new file rather than wrapping it in a closure. Someone implementing a flame library will also need to write the pooling, monitoring, and remote communication bits. We get a lot for free in Elixir on the distributed messaging and monitoring side. The process placement stuff is also really only applicable to Elixir. Hope that helps!
- jrmiii 3y ago
  > functions with captured variable serialization Can't wait for the deep dive on how that works
  - azurelake 3y ago
    Probably not too much to say that’s specific to FLAME. Closures are serializable and can be sent as messages to actors on the BEAM with a few caveats. From a quick look at the code, this looks the magic line: https://github.com/phoenixframework/flame/blob/main/lib/flame/runner.ex#L407-L408 https://github.com/phoenixframework/flame/blob/main/lib/flam...
  - quaunaut 3y ago
    That's just standard erlang/elixir- because all values are immutable, when a new anonymous function is defined it copies the current value of the external variables into it. You can do it right now even without Flame, just by opening two Elixir nodes, then it's as simple as ```elixir iex(first_node@localhost)> name = "Santa" iex(first_node@localhost)> Node.spawn_link(:other_node@localhost, fn -> IO.puts "Hello #{name}" end) Hello Santa #PID<1337.42.0> ``` Note that while the string interpolation and `IO.puts` was run on `other_node@localhost`, it still did stdout from the first node- this is because it was the one that called `Node.spawn_link`, making it the 'group leader'. Outside of which stdout it went to, all the work was done in the other node.
- crabmusket 3y ago
  JS kind of has inter process built in with Web Workers and the channel messaging API- I wonder whether it'd be possible to essentially create a "FLAME Worker" with the same API as a web worker but backed by distributed execution!
hinkley 3y ago
A pattern I see over and over, which has graduated to somewhere between a theorem and a law, is that motivated developers can make just about any process or architecture work for about 18 months. By the time things get bad, it's almost time to find a new job, especially if the process was something you introduced a year or more into your tenure and are now regretting. I've seen it with a handful of bad bosses, at least half a dozen times with (shitty) 'unit testing', scrum, you name it. But what I don't know is how many people are mentally aware of the sources of discomfort they feel at work, instead of a more nebulous "it's time to move on". I certainly get a lot of pushback trying to name uncomfortable things (and have a lot less bad feelings about it now that I've read Good to Great). Nobody wants to say, "Oh look, the consequences of my actions." The people materially responsible for the Rube Goldberg machine I help maintain were among the first to leave. The captain of that ship asked a coworker of mine if he thought it would be a good idea to open source our engine. He responded that nobody would want to use our system when the wheels it reinvented already exist (and are better). That guy was gone within three to four months, under his own steam.
- antod 3y ago
  That's why I'm always wary of people who hardly ever seem to stay anywhere more than a couple of years. There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle. Understanding how tech debt comes to be, what tradeoffs were involved and how they came to bite later. Which ideas turned out to be bad in hindsight through the lens of the people making them at the time. Rather than just painting the previous crowd as incompetent while simultaneously making worse decisions you'll never experience the consequences of. Moving on every 18-24 months leaves you with a potentially false impression of your own skills/wisdom.
  - jvans 3y ago
    > There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle. This is so true. It's extremely enlightening to watch a design go from design docs to implementation and then finally the maintenance phase. A lot of problems that could happen never do and some unexpected ones pop up along the way.
icedchai 3y ago
I couldn't even stand having a dozen lambdas. The app was originally built by someone who didn't think much about maintenance or deployment. Code was copy-pasted all over the place. Eventually, we moved to a "fat lambda" monolith where a single lambda serves multiple endpoints.
viraptor 3y ago
> that you wish you had just gone the monolith route Going from hundreds of lambdas to a monolith is overreacting to one extreme by going the other one. There's a whole spectrum of possible ways to split a project in useful ways, which simplify development and maintenance.
- Spivak 3y ago
  Anything in between is all the downsides of both approaches. Once you have the flow for deploying always running hot application(s) with autoscaling the benefits of lambda are basically gone. Low volume scale to zero is just another route. No 15 minute limit, no having to marshal data through other AWS services because that's all lambda talks to, no more eventbridge for cron, no more payload size limits and having to use S3 as buffer, no more network requests between different parts of the same logical app, code deploys are atomic you're either at v1.x or v1.x+1 but never some in-between state. I really do like Lambda but once you're at the spend where it's the same as some dedicated always-on compute the value drops off.
p10jkle 3y ago
I'm working on something that I think might solve the problem in any language (currently have an sdk for typescript, and java in the works). You can avoid splitting an application into 100s of small short-running chunks if you can write normal service-orientated code, where lambdas can call each other. But this isn't possible without paying for all that time waiting around. If the Lambdas can pause execution while they are blocked on IO, it solves the problem. So I think durable execution might be the answer! I've been working on a blog post to show this off for the last couple of weeks: https://restate.dev/blog/suspendable-functions-make-lambda-the-perfect-application-platform/ https://restate.dev/blog/suspendable-functions-make-lambda-t...
Rapzid 3y ago
You can monolith on lamba if you don't care too much about cold starts or can manage it.. Put another way, you can monolith and minimize spend on AWS; it's not either or. I'm using asp.net these days and even a chunky app published ready to run with optimized ef models starts relatively quickly.
randall 3y ago
I don't get why you'd have a 100+ lambda function app... i can see purpose built lambdas (ie we have one for "graphql" and "frontend" and a few backend services) but unless you're at Meta size, why would you have 100 lambdas? Do you have 100 teams?

chrismccord 3y ago

I talk about FLAME outside elixir in one one of the sections in the blog. The tldr; is it's a generally applicable pattern for languages with a reasonable concurrency model. You likely won't get all the ergonomics that we get for free like functions with captured variable serialization, but you can probably get 90% of the way there in something like js, where you can move your modular execution to a new file rather than wrapping it in a closure. Someone implementing a flame library will also need to write the pooling, monitoring, and remote communication bits. We get a lot for free in Elixir on the distributed messaging and monitoring side. The process placement stuff is also really only applicable to Elixir. Hope that helps!

jrmiii 3y ago
> functions with captured variable serialization Can't wait for the deep dive on how that works
- azurelake 3y ago
  Probably not too much to say that’s specific to FLAME. Closures are serializable and can be sent as messages to actors on the BEAM with a few caveats. From a quick look at the code, this looks the magic line: https://github.com/phoenixframework/flame/blob/main/lib/flame/runner.ex#L407-L408 https://github.com/phoenixframework/flame/blob/main/lib/flam...
- quaunaut 3y ago
  That's just standard erlang/elixir- because all values are immutable, when a new anonymous function is defined it copies the current value of the external variables into it. You can do it right now even without Flame, just by opening two Elixir nodes, then it's as simple as ```elixir iex(first_node@localhost)> name = "Santa" iex(first_node@localhost)> Node.spawn_link(:other_node@localhost, fn -> IO.puts "Hello #{name}" end) Hello Santa #PID<1337.42.0> ``` Note that while the string interpolation and `IO.puts` was run on `other_node@localhost`, it still did stdout from the first node- this is because it was the one that called `Node.spawn_link`, making it the 'group leader'. Outside of which stdout it went to, all the work was done in the other node.
crabmusket 3y ago
JS kind of has inter process built in with Web Workers and the channel messaging API- I wonder whether it'd be possible to essentially create a "FLAME Worker" with the same API as a web worker but backed by distributed execution!

hinkley 3y ago

A pattern I see over and over, which has graduated to somewhere between a theorem and a law, is that motivated developers can make just about any process or architecture work for about 18 months. By the time things get bad, it's almost time to find a new job, especially if the process was something you introduced a year or more into your tenure and are now regretting. I've seen it with a handful of bad bosses, at least half a dozen times with (shitty) 'unit testing', scrum, you name it. But what I don't know is how many people are mentally aware of the sources of discomfort they feel at work, instead of a more nebulous "it's time to move on". I certainly get a lot of pushback trying to name uncomfortable things (and have a lot less bad feelings about it now that I've read Good to Great). Nobody wants to say, "Oh look, the consequences of my actions." The people materially responsible for the Rube Goldberg machine I help maintain were among the first to leave. The captain of that ship asked a coworker of mine if he thought it would be a good idea to open source our engine. He responded that nobody would want to use our system when the wheels it reinvented already exist (and are better). That guy was gone within three to four months, under his own steam.

antod 3y ago
That's why I'm always wary of people who hardly ever seem to stay anywhere more than a couple of years. There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle. Understanding how tech debt comes to be, what tradeoffs were involved and how they came to bite later. Which ideas turned out to be bad in hindsight through the lens of the people making them at the time. Rather than just painting the previous crowd as incompetent while simultaneously making worse decisions you'll never experience the consequences of. Moving on every 18-24 months leaves you with a potentially false impression of your own skills/wisdom.
- jvans 3y ago
  > There's valuable learning (and empathy too) in having to see your own decisions and creations through their whole lifecycle. This is so true. It's extremely enlightening to watch a design go from design docs to implementation and then finally the maintenance phase. A lot of problems that could happen never do and some unexpected ones pop up along the way.

icedchai 3y ago

I couldn't even stand having a dozen lambdas. The app was originally built by someone who didn't think much about maintenance or deployment. Code was copy-pasted all over the place. Eventually, we moved to a "fat lambda" monolith where a single lambda serves multiple endpoints.

viraptor 3y ago

> that you wish you had just gone the monolith route Going from hundreds of lambdas to a monolith is overreacting to one extreme by going the other one. There's a whole spectrum of possible ways to split a project in useful ways, which simplify development and maintenance.

Spivak 3y ago
Anything in between is all the downsides of both approaches. Once you have the flow for deploying always running hot application(s) with autoscaling the benefits of lambda are basically gone. Low volume scale to zero is just another route. No 15 minute limit, no having to marshal data through other AWS services because that's all lambda talks to, no more eventbridge for cron, no more payload size limits and having to use S3 as buffer, no more network requests between different parts of the same logical app, code deploys are atomic you're either at v1.x or v1.x+1 but never some in-between state. I really do like Lambda but once you're at the spend where it's the same as some dedicated always-on compute the value drops off.

p10jkle 3y ago

I'm working on something that I think might solve the problem in any language (currently have an sdk for typescript, and java in the works). You can avoid splitting an application into 100s of small short-running chunks if you can write normal service-orientated code, where lambdas can call each other. But this isn't possible without paying for all that time waiting around. If the Lambdas can pause execution while they are blocked on IO, it solves the problem. So I think durable execution might be the answer! I've been working on a blog post to show this off for the last couple of weeks: https://restate.dev/blog/suspendable-functions-make-lambda-the-perfect-application-platform/ https://restate.dev/blog/suspendable-functions-make-lambda-t...

Rapzid 3y ago

You can monolith on lamba if you don't care too much about cold starts or can manage it.. Put another way, you can monolith and minimize spend on AWS; it's not either or. I'm using asp.net these days and even a chunky app published ready to run with optimized ef models starts relatively quickly.

randall 3y ago

I don't get why you'd have a 100+ lambda function app... i can see purpose built lambdas (ie we have one for "graphql" and "frontend" and a few backend services) but unless you're at Meta size, why would you have 100 lambdas? Do you have 100 teams?