5 ms·

Lambda scales faster if you really do need that. For instance, imagine bursts of 100k requests. Cold start on Lambda is going to be lower than you can autoscale

by nijave 2y ago

Lambda scales faster if you really do need that. For instance, imagine bursts of 100k requests. Cold start on Lambda is going to be lower than you can autoscale something else.

ndriscoll 2y ago
What actually happens is you hit the concurrency limit and return 99k throttling errors, or your lambdas will try to fire up 100k database connections and will again probably just give you 99k-100k errors. Meanwhile a 1 CPU core container would be done with your burst in a few seconds. What in the world needs to handle random bursts from 0 to 100k requests all at once though? I struggle to imagine anything. Lambda might be a decent fit for bursty CPU-intensive work that doesn't need to do IO and can take advantage of multiple cores for a single request, which is not many web applications.
- chuckadams 2y ago
  If you go from 0 to 100K legit requests in the same instant, any sane architecture will ramp up autoscaling, but not so instantly that it tries to serve every last one of them in that moment. Most of them get throttled. A well-behaved client will back off and retry, and a badly-behaved client can go pound sand. But reality is, if you crank the max instances knob to 100k and they all open DB connections, your DB had better handle 100k connections.
  - ndriscoll 2y ago
    A sane architecture would be running your application on something that has at least the resources of a phone (e.g. 8+ GB RAM), in which case it should just buffer 100k connections without issue if that's what you need/want it to do. A sane application framework has connection pooling to the database built in, so the 100k requests would share ~16-32 connections and your developers never have to think about such things.
    - chuckadams 2y ago
      I'd call pooling part of "the DB". "DB Layer" if you must, or the interface to it, whatever. Anyway, AWS has RDS Proxy, which held up pretty well against my (ad hoc and unscientific) load tests. But if you're actually trying to handle 100K DB requests in flight at once, your DB layer probably has some distributed architecture going on already.
      - ndriscoll 2y ago
        If you're using RDS proxy, now you're not scaling to zero, and you still can't handle 100k burst requests because lambda can't do that. So why not use a normal application architecture which actually can handle bursts no problem and doesn't need a distributed database? Lambda could be a compelling offering for many use cases if they made it so that you could set concurrency on each invocation. e.g. only spin up one invocation if you have fewer then 1k requests in flight, and let that invocation process them concurrently. But as long as it can only do 1 request at a time per invocation, it's just a vastly worse version of spinning up 1 process per request, which we moved away from because of how poorly it scales.
        nijave 2y ago
        Lambda can scale to 10k in 1 minute https://aws.amazon.com/blogs/aws/aws-lambda-functions-now-scale-12-times-faster-when-handling-high-volume-requests/ https://aws.amazon.com/blogs/aws/aws-lambda-functions-now-sc... If your response time is 100ms, that's 100k requests in 1 minute. Lambda runs your code in a VM that's kept hot so repeated invocations aren't launching processes. AWS is eating the cost of keeping the infra idle for you (arguably passing it on).
        ndriscoll 2y ago
        A normal application can scale to 10k concurrent requests as fast as they come in (i.e. a fraction of a second). Even at 16kB/request, that's a little over 160 MB of memory. That's the point: a socket and a couple objects per request scales way better than 1 process/request which scales way better than 1 VM per request, regardless of how hot you keep the VM. Serving 10k concurrent connections/requests was an interesting problem in 1999. People were doing 10M on one server 10 years ago[0]. Lambda is traveling back in time 30 years. [0] https://migratorydata.com/blog/migratorydata-solved-the-c10m-problem/ https://migratorydata.com/blog/migratorydata-solved-the-c10m...
- nijave 2y ago
  You'd obviously need to make sure the persistence pieces and downstream components can also handle this. You could dump items onto a queue or utilize key value like Dynamo A 1 CPU container isn't going to handle that many "app" requests unless you have a trivial (almost no logic) or highly optimized (like c static web server) app Lambda apps don't need to take advantage of multiple cores since you get a guaranteed fractional core per request One example is retail fire sales. I interviewed with a company that had this exact use case. They produced anti bot software for limited product releases and needed to handle this load (they required their customers submit a request ahead of time to pre scale) Also useful with telemetry systems where you might get a burst in logs or metrics and want to consume as fast as possible to avoid dropped data and buffering on the source but can dump to a queue for async processing
  - ndriscoll 2y ago
    In my experience, real world applications mostly do deal with basically trivial requests (e.g. simple CRUD with some auth checks). The starting point I'm used to for multiple web frameworks is in the thousands of requests/second. Something like telemetry is definitely trivial, and should be easy to batch to get into the 10s of thousands of requests/second. The 25 ms request you mentioned in another comment is what I'd categorize as extremely CPU intensive.

by nijave 2y ago

Lambda scales faster if you really do need that. For instance, imagine bursts of 100k requests. Cold start on Lambda is going to be lower than you can autoscale something else.

ndriscoll 2y ago

What actually happens is you hit the concurrency limit and return 99k throttling errors, or your lambdas will try to fire up 100k database connections and will again probably just give you 99k-100k errors. Meanwhile a 1 CPU core container would be done with your burst in a few seconds. What in the world needs to handle random bursts from 0 to 100k requests all at once though? I struggle to imagine anything. Lambda might be a decent fit for bursty CPU-intensive work that doesn't need to do IO and can take advantage of multiple cores for a single request, which is not many web applications.

chuckadams 2y ago
If you go from 0 to 100K legit requests in the same instant, any sane architecture will ramp up autoscaling, but not so instantly that it tries to serve every last one of them in that moment. Most of them get throttled. A well-behaved client will back off and retry, and a badly-behaved client can go pound sand. But reality is, if you crank the max instances knob to 100k and they all open DB connections, your DB had better handle 100k connections.
- ndriscoll 2y ago
  A sane architecture would be running your application on something that has at least the resources of a phone (e.g. 8+ GB RAM), in which case it should just buffer 100k connections without issue if that's what you need/want it to do. A sane application framework has connection pooling to the database built in, so the 100k requests would share ~16-32 connections and your developers never have to think about such things.
  - chuckadams 2y ago
    I'd call pooling part of "the DB". "DB Layer" if you must, or the interface to it, whatever. Anyway, AWS has RDS Proxy, which held up pretty well against my (ad hoc and unscientific) load tests. But if you're actually trying to handle 100K DB requests in flight at once, your DB layer probably has some distributed architecture going on already.
    - ndriscoll 2y ago
      If you're using RDS proxy, now you're not scaling to zero, and you still can't handle 100k burst requests because lambda can't do that. So why not use a normal application architecture which actually can handle bursts no problem and doesn't need a distributed database? Lambda could be a compelling offering for many use cases if they made it so that you could set concurrency on each invocation. e.g. only spin up one invocation if you have fewer then 1k requests in flight, and let that invocation process them concurrently. But as long as it can only do 1 request at a time per invocation, it's just a vastly worse version of spinning up 1 process per request, which we moved away from because of how poorly it scales.
      - nijave 2y ago
        Lambda can scale to 10k in 1 minute https://aws.amazon.com/blogs/aws/aws-lambda-functions-now-scale-12-times-faster-when-handling-high-volume-requests/ https://aws.amazon.com/blogs/aws/aws-lambda-functions-now-sc... If your response time is 100ms, that's 100k requests in 1 minute. Lambda runs your code in a VM that's kept hot so repeated invocations aren't launching processes. AWS is eating the cost of keeping the infra idle for you (arguably passing it on).
        ndriscoll 2y ago
        A normal application can scale to 10k concurrent requests as fast as they come in (i.e. a fraction of a second). Even at 16kB/request, that's a little over 160 MB of memory. That's the point: a socket and a couple objects per request scales way better than 1 process/request which scales way better than 1 VM per request, regardless of how hot you keep the VM. Serving 10k concurrent connections/requests was an interesting problem in 1999. People were doing 10M on one server 10 years ago[0]. Lambda is traveling back in time 30 years. [0] https://migratorydata.com/blog/migratorydata-solved-the-c10m-problem/ https://migratorydata.com/blog/migratorydata-solved-the-c10m...
nijave 2y ago
You'd obviously need to make sure the persistence pieces and downstream components can also handle this. You could dump items onto a queue or utilize key value like Dynamo A 1 CPU container isn't going to handle that many "app" requests unless you have a trivial (almost no logic) or highly optimized (like c static web server) app Lambda apps don't need to take advantage of multiple cores since you get a guaranteed fractional core per request One example is retail fire sales. I interviewed with a company that had this exact use case. They produced anti bot software for limited product releases and needed to handle this load (they required their customers submit a request ahead of time to pre scale) Also useful with telemetry systems where you might get a burst in logs or metrics and want to consume as fast as possible to avoid dropped data and buffering on the source but can dump to a queue for async processing
- ndriscoll 2y ago
  In my experience, real world applications mostly do deal with basically trivial requests (e.g. simple CRUD with some auth checks). The starting point I'm used to for multiple web frameworks is in the thousands of requests/second. Something like telemetry is definitely trivial, and should be easy to batch to get into the 10s of thousands of requests/second. The 25 ms request you mentioned in another comment is what I'd categorize as extremely CPU intensive.