3 ms·

I was really excited when reading this, but realized the lack of a faster "warm" start makes this less ideal for my highly latency-sensitive use case on Lambda.

I was really excited when reading this, but realized the lack of a faster "warm" start makes this less ideal for my highly latency-sensitive use case on Lambda. Lambdas start much faster than 300ms when warm IME, and I'm hoping with enough sustained traffic (be it real or artificial), most requests will be warm. I'd love to be able to supply some kind of memory snapshot in addition to the docker image to cut down on cold starts. Probably blocked on snapshot support in Firecracker according to another thread? Eagerly awaiting this since it could make Fly Machine the best of both worlds! Not a fan of how Lambda makes me scale memory and compute in tandem, when my use case benefits so much more from compute than memory. I basically have to pay for 2+ gigs I'm never going to use to get the compute performance I want. Makes 0 sense.

nicoburns 4y ago
> Lambdas start much faster than 300ms when warm IME. My understanding is that Lambdas aren't ever really truly warm unless you have a completely steady traffic level. The first request in a while will hit the latency spike. But so will every increase in concurrency. So if you were serving 2 req/sec, and a 3rd concurrent visitor comes along then they will also get a cold start. If you have a low-latency use case then fly.io's regular VMs are much better than either this or lambda. You get permanently running VM (the smallest of which is $2/month for 256mb/RAM), which can serve more than one request all by itself and will auto-scale with traffic.
- lewisl9029 4y ago
  I do actually already use Fly for pretty much everything else. For this use case though, I forgot to mention that it needs _much_ faster autoscaling than what Fly's regular VMs offer, with unbounded concurrency, and not ideal to run concurrently in a single VM due to each request being compute heavy and needing full isolation from each other since they run arbitrary customer code. It's true that with Lambda, some amount of cold starts are probably inevitable with extreme spikes in traffic. But I'm hoping to mitigate most of that by sending artificial concurrent traffic on a schedule to keep a decent buffer of warmed up Lambdas above the current real traffic level. Still to be seen if that plan works out in practice.
  - nicoburns 4y ago
    Hmm... If you're willing to run the scheduler and/or something sending artificial traffic yourself (and are willing to pay for a few warm instances), then it seems like you might be willing to get something very low-latency with these Fly Machines. You could maintain a pool of a few booted and ready to go (but idle, waiting for a request to come in) at all times. When a request comes in you pass that off to an already-warm machine and boot a new one ready for a later request. Though I think ultimately it's going to be impossible to have all 3 of: - Ability to deal with large traffic spikes - Low-latency - No provisioned resources
    - lewisl9029 4y ago
      For my use case, the main benefit of using manually pre-warmed Lambda instances over Fly Machines is the fact that "warm" Lambdas cost practically nothing (only have to pay for the warming events that are billed for milliseconds every few minutes). :) This is why I want to see a similar mode of operation for Fly Machines, possibly through memory snapshots, so I can manually provide a "warm" suspend state for it to unsuspend into. In fact this would be even better than the lambda model since there would be _no_ cold starts.

nicoburns 4y ago

> Lambdas start much faster than 300ms when warm IME. My understanding is that Lambdas aren't ever really truly warm unless you have a completely steady traffic level. The first request in a while will hit the latency spike. But so will every increase in concurrency. So if you were serving 2 req/sec, and a 3rd concurrent visitor comes along then they will also get a cold start. If you have a low-latency use case then fly.io's regular VMs are much better than either this or lambda. You get permanently running VM (the smallest of which is $2/month for 256mb/RAM), which can serve more than one request all by itself and will auto-scale with traffic.

lewisl9029 4y ago
I do actually already use Fly for pretty much everything else. For this use case though, I forgot to mention that it needs _much_ faster autoscaling than what Fly's regular VMs offer, with unbounded concurrency, and not ideal to run concurrently in a single VM due to each request being compute heavy and needing full isolation from each other since they run arbitrary customer code. It's true that with Lambda, some amount of cold starts are probably inevitable with extreme spikes in traffic. But I'm hoping to mitigate most of that by sending artificial concurrent traffic on a schedule to keep a decent buffer of warmed up Lambdas above the current real traffic level. Still to be seen if that plan works out in practice.
- nicoburns 4y ago
  Hmm... If you're willing to run the scheduler and/or something sending artificial traffic yourself (and are willing to pay for a few warm instances), then it seems like you might be willing to get something very low-latency with these Fly Machines. You could maintain a pool of a few booted and ready to go (but idle, waiting for a request to come in) at all times. When a request comes in you pass that off to an already-warm machine and boot a new one ready for a later request. Though I think ultimately it's going to be impossible to have all 3 of: - Ability to deal with large traffic spikes - Low-latency - No provisioned resources
  - lewisl9029 4y ago
    For my use case, the main benefit of using manually pre-warmed Lambda instances over Fly Machines is the fact that "warm" Lambdas cost practically nothing (only have to pay for the warming events that are billed for milliseconds every few minutes). :) This is why I want to see a similar mode of operation for Fly Machines, possibly through memory snapshots, so I can manually provide a "warm" suspend state for it to unsuspend into. In fact this would be even better than the lambda model since there would be _no_ cold starts.