8 ms·

We had to abandon Lambda due to cold starts. Any news if that's resolved?

by watty 8y ago

jedberg 8y ago
You can generally resolve it yourself by poking seldom used functions to keep them hot. But no, they haven’t provided a solution to cold start (unless you consider ec2 or fargate a solution).
- antimatter 8y ago
  > You can generally resolve it yourself by poking seldom used functions to keep them hot. We've tried this and it helps somewhat but when AWS attempts to scale your function based on load, cold starts re-appear. We've moved away from Lambdas where a dependable response time is required.
  - Touche 8y ago
    Can you explain more about why this is? If you are experiencing cold starts it means that function is not used very often. If it's not used very often that likely means it's not user facing (or something less important like a Terms of Service page). If that's the case, why do you need instant response times?
    - mirceal 8y ago
      that’s not entirely true. while your warm lambdas can and will take the traffic it your traffic ramps up, additional lambda instances will be spun up. you will pay cold start prices as they are spinning up. so, even if you have a heavily used lambda fn, depending on the traffic your p99 will still look pretty bad and you will not be able to guarantee that all requests will be processes in x ms or less.
    - count 8y ago
      No, that's not what it means. If you have high concurrent execution, you get 'cold start' every time the underlying service 'scales out' to support more. The MORE you use lambda concurrently, the more you hit the cold start issue. Granted, it's just for that one cold start execution per-scale node (and they could probably just optionally pre-fire to warm things in that instance, like with a cache), but it's definitely there horizontally.
      - deegles 8y ago
        Also, containers are restarted periodically (~4 hours) so even if you have very steady traffic you'll see cold starts regularly.
      - spullara 8y ago
        I really with they would add an init() callback that is called on cold start but before any traffic is sent to your lambda. It wouldn't help when there are no lambdas running but it could be useful when things are scaling up, especially if you can ask for additional concurrency above the actual concurrency necessary for spikes.
        redisman 8y ago
        More lifecycle events please! I'd love a onThawed and onFrozen or something so I can kill the DB connections neatly
        jacobr1 8y ago
        You can already do this, I think. Just put this logic in your application's static initialization.
        spullara 8y ago
        I don't think so. When it spins it up, the request is already in flight. Otherwise this would have been solved by everyone but instead everyone sees terrible cold start times.
    - hn_throwaway_99 8y ago
      This is along the lines of what the other responses to this comment have said, but https://hackernoon.com/im-afraid-you-re-thinking-about-aws-lambda-cold-starts-all-wrong-7d907f278a4f https://hackernoon.com/im-afraid-you-re-thinking-about-aws-l... gives a very detailed overview. It's titled "I'm afraid you’re thinking about AWS Lambda cold starts all wrong", because the way you're thinking about cold start times is common (and wrong).
- jvanvleet 8y ago
  https://github.com/awslabs/aws-lambda-cpp https://github.com/awslabs/aws-lambda-cpp makes claims of millisecond cold start.
  - hn_throwaway_99 8y ago
    I'm wondering how that's even possible if it includes the time for downloading your code from S3. I.e. normal cold starts (as I understand it) involve fetching the code from S3 to install on a VM. Perhaps they aren't including that time when claiming single milli cold start times?
  - nostrebored 8y ago
    That's not for in-VPC functions, although if the underlying instance changes with firecracker migration users might see ENI start improvements. Currently your ENI usage is roughly floor(3GB/Memory Allocation)*(Concurrent Executions). If the 3GB changes users will see huge gains as each ENI creation can take around 9s.
    - k__ 8y ago
      Why would you use VPC functions?
      - Wintereise 8y ago
        Mostly due to being forced. For example: serverless aurora forces a VPC on you. With delays as bad as 8+ seconds for ENI attachment, this is a significant holdback (https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-lat...)
        k__ 8y ago
        Ah okay, I never thought about using an SQL DB.
        zwily 8y ago
        I believe you can now use Aurora Serverless outside of a VPC by using the new Data API endpoint.
otterley 8y ago
Cold start performance is improving significantly as the Lambda fleets migrate to Firecracker technology: https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...
- saosebastiao 8y ago
  Cold starts for the VM are only part of the problem. If you're on a JITed runtime, a cold start also means compilation and optimization. It would be nice if they had ways to cache the JITed machine code so they could start back up with it already compiled and optimized.
  - deleted 8y ago
    [deleted]
  - kondro 8y ago
    Modern lambdas can be paused for 4+ hours before experiencing cold starts (depending on a lot of variables). AWS is making continual improvements in this area.
- hn_throwaway_99 8y ago
  Anyone know if there have been any improvements to cold start times for Lambdas in a VPC? That was the absolute death knell for us. If you're using Lambdas as a service backend for mobile/web apps, it's extremely common those Lambdas will be talking to a DB, and any decent security practice would require that DB to be in a VPC. Cold starts for Lambdas in a VPC could be on the order of 8-10 seconds: https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-lat...
  - staticassertion 8y ago
    Agreed! I'm much more concerned with VPC performance - I don't have a single lambda outside of a VPC. Firecracker is extremely cool, and I'm very glad to see the improved perf at the VM level, but that's not my bottleneck. Thankfully, in my case, I have a very steady flow of data so I don't expect too many cold starts.
jfoldager 8y ago
Out of curiosity: What is the response time on cold starts?
- nostrebored 8y ago
  In VPC probably around 10s if there are no ENIs available
zwily 8y ago
I was really hoping they would announce pre-warming for all Lambdas. :(
k__ 8y ago
As far as I know they aren't an issue of AppSync, the managed GraphQL service of AWS. You can basically replace all HTTP-API Lambdas with it, because it allows direct access to DynamoDB.

We had to abandon Lambda due to cold starts. Any news if that's resolved?

by watty 8y ago

We had to abandon Lambda due to cold starts. Any news if that's resolved?

You can generally resolve it yourself by poking seldom used functions to keep them hot. But no, they haven’t provided a solution to cold start (unless you consider ec2 or fargate a solution).

antimatter 8y ago
> You can generally resolve it yourself by poking seldom used functions to keep them hot. We've tried this and it helps somewhat but when AWS attempts to scale your function based on load, cold starts re-appear. We've moved away from Lambdas where a dependable response time is required.
- Touche 8y ago
  Can you explain more about why this is? If you are experiencing cold starts it means that function is not used very often. If it's not used very often that likely means it's not user facing (or something less important like a Terms of Service page). If that's the case, why do you need instant response times?
  - mirceal 8y ago
    that’s not entirely true. while your warm lambdas can and will take the traffic it your traffic ramps up, additional lambda instances will be spun up. you will pay cold start prices as they are spinning up. so, even if you have a heavily used lambda fn, depending on the traffic your p99 will still look pretty bad and you will not be able to guarantee that all requests will be processes in x ms or less.
  - count 8y ago
    No, that's not what it means. If you have high concurrent execution, you get 'cold start' every time the underlying service 'scales out' to support more. The MORE you use lambda concurrently, the more you hit the cold start issue. Granted, it's just for that one cold start execution per-scale node (and they could probably just optionally pre-fire to warm things in that instance, like with a cache), but it's definitely there horizontally.
    - deegles 8y ago
      Also, containers are restarted periodically (~4 hours) so even if you have very steady traffic you'll see cold starts regularly.
    - spullara 8y ago
      I really with they would add an init() callback that is called on cold start but before any traffic is sent to your lambda. It wouldn't help when there are no lambdas running but it could be useful when things are scaling up, especially if you can ask for additional concurrency above the actual concurrency necessary for spikes.
      - redisman 8y ago
        More lifecycle events please! I'd love a onThawed and onFrozen or something so I can kill the DB connections neatly
      - jacobr1 8y ago
        You can already do this, I think. Just put this logic in your application's static initialization.
        spullara 8y ago
        I don't think so. When it spins it up, the request is already in flight. Otherwise this would have been solved by everyone but instead everyone sees terrible cold start times.
  - hn_throwaway_99 8y ago
    This is along the lines of what the other responses to this comment have said, but https://hackernoon.com/im-afraid-you-re-thinking-about-aws-lambda-cold-starts-all-wrong-7d907f278a4f https://hackernoon.com/im-afraid-you-re-thinking-about-aws-l... gives a very detailed overview. It's titled "I'm afraid you’re thinking about AWS Lambda cold starts all wrong", because the way you're thinking about cold start times is common (and wrong).
jvanvleet 8y ago
https://github.com/awslabs/aws-lambda-cpp https://github.com/awslabs/aws-lambda-cpp makes claims of millisecond cold start.
- hn_throwaway_99 8y ago
  I'm wondering how that's even possible if it includes the time for downloading your code from S3. I.e. normal cold starts (as I understand it) involve fetching the code from S3 to install on a VM. Perhaps they aren't including that time when claiming single milli cold start times?
- nostrebored 8y ago
  That's not for in-VPC functions, although if the underlying instance changes with firecracker migration users might see ENI start improvements. Currently your ENI usage is roughly floor(3GB/Memory Allocation)*(Concurrent Executions). If the 3GB changes users will see huge gains as each ENI creation can take around 9s.
  - k__ 8y ago
    Why would you use VPC functions?
    - Wintereise 8y ago
      Mostly due to being forced. For example: serverless aurora forces a VPC on you. With delays as bad as 8+ seconds for ENI attachment, this is a significant holdback (https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-lat...)
      - k__ 8y ago
        Ah okay, I never thought about using an SQL DB.
      - zwily 8y ago
        I believe you can now use Aurora Serverless outside of a VPC by using the new Data API endpoint.

otterley 8y ago

Cold start performance is improving significantly as the Lambda fleets migrate to Firecracker technology: https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/ https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...

saosebastiao 8y ago
Cold starts for the VM are only part of the problem. If you're on a JITed runtime, a cold start also means compilation and optimization. It would be nice if they had ways to cache the JITed machine code so they could start back up with it already compiled and optimized.
- deleted 8y ago
  [deleted]
- kondro 8y ago
  Modern lambdas can be paused for 4+ hours before experiencing cold starts (depending on a lot of variables). AWS is making continual improvements in this area.
hn_throwaway_99 8y ago
Anyone know if there have been any improvements to cold start times for Lambdas in a VPC? That was the absolute death knell for us. If you're using Lambdas as a service backend for mobile/web apps, it's extremely common those Lambdas will be talking to a DB, and any decent security practice would require that DB to be in a VPC. Cold starts for Lambdas in a VPC could be on the order of 8-10 seconds: https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-lat...
- staticassertion 8y ago
  Agreed! I'm much more concerned with VPC performance - I don't have a single lambda outside of a VPC. Firecracker is extremely cool, and I'm very glad to see the improved perf at the VM level, but that's not my bottleneck. Thankfully, in my case, I have a very steady flow of data so I don't expect too many cold starts.

jfoldager 8y ago

Out of curiosity: What is the response time on cold starts?

nostrebored 8y ago
In VPC probably around 10s if there are no ENIs available

zwily 8y ago

I was really hoping they would announce pre-warming for all Lambdas. :(

k__ 8y ago

As far as I know they aren't an issue of AppSync, the managed GraphQL service of AWS. You can basically replace all HTTP-API Lambdas with it, because it allows direct access to DynamoDB.