4 ms·

I do actually already use Fly for pretty much everything else. For this use case though, I forgot to mention that it needs _much_ faster autoscaling than what

I do actually already use Fly for pretty much everything else. For this use case though, I forgot to mention that it needs _much_ faster autoscaling than what Fly's regular VMs offer, with unbounded concurrency, and not ideal to run concurrently in a single VM due to each request being compute heavy and needing full isolation from each other since they run arbitrary customer code. It's true that with Lambda, some amount of cold starts are probably inevitable with extreme spikes in traffic. But I'm hoping to mitigate most of that by sending artificial concurrent traffic on a schedule to keep a decent buffer of warmed up Lambdas above the current real traffic level. Still to be seen if that plan works out in practice.

nicoburns 4y ago
Hmm... If you're willing to run the scheduler and/or something sending artificial traffic yourself (and are willing to pay for a few warm instances), then it seems like you might be willing to get something very low-latency with these Fly Machines. You could maintain a pool of a few booted and ready to go (but idle, waiting for a request to come in) at all times. When a request comes in you pass that off to an already-warm machine and boot a new one ready for a later request. Though I think ultimately it's going to be impossible to have all 3 of: - Ability to deal with large traffic spikes - Low-latency - No provisioned resources
- lewisl9029 4y ago
  For my use case, the main benefit of using manually pre-warmed Lambda instances over Fly Machines is the fact that "warm" Lambdas cost practically nothing (only have to pay for the warming events that are billed for milliseconds every few minutes). :) This is why I want to see a similar mode of operation for Fly Machines, possibly through memory snapshots, so I can manually provide a "warm" suspend state for it to unsuspend into. In fact this would be even better than the lambda model since there would be _no_ cold starts.

nicoburns 4y ago

Hmm... If you're willing to run the scheduler and/or something sending artificial traffic yourself (and are willing to pay for a few warm instances), then it seems like you might be willing to get something very low-latency with these Fly Machines. You could maintain a pool of a few booted and ready to go (but idle, waiting for a request to come in) at all times. When a request comes in you pass that off to an already-warm machine and boot a new one ready for a later request. Though I think ultimately it's going to be impossible to have all 3 of: - Ability to deal with large traffic spikes - Low-latency - No provisioned resources

lewisl9029 4y ago
For my use case, the main benefit of using manually pre-warmed Lambda instances over Fly Machines is the fact that "warm" Lambdas cost practically nothing (only have to pay for the warming events that are billed for milliseconds every few minutes). :) This is why I want to see a similar mode of operation for Fly Machines, possibly through memory snapshots, so I can manually provide a "warm" suspend state for it to unsuspend into. In fact this would be even better than the lambda model since there would be _no_ cold starts.