5 ms·

> It then finds or boots a new copy of our entire application and runs the function there. So for each “Flame.call” it begins a whole new app process and copie

by gitgud 3y ago

> It then finds or boots a new copy of our entire application and runs the function there. So for each “Flame.call” it begins a whole new app process and copies the execution context in? A very simple solution to scaling, but I’d imagine this would have some disadvantages… Adding 10ms to the app startup time, adds 10ms to every “Flame.call” part of the application too… same with memory I suppose I guess these concerns just need to be consider when using this system

chrismccord 3y ago
The FLAME.Pool discussed later in the post addresses this. Runners are pooled and remain configurable hot for whatever time you want before idling down. Under load you are rarely paying the cold start time because the pool is already hot. We are also adding more sophisticated pool growth techniques to the Elixir library next so you also avoid hitting an at capacity runner and cold starting one. For hot runners, the only overhead is the latency between the parent and child, which should be the same datacenter so 1ms or sub 1ms.
- bo0tzz 3y ago
  Currently the per-runner concurrency is limited by a fixed number. Have you thought about approaches that instead base this on resource usage, so that runners can be used optimally?
  - chrismccord 3y ago
    Yes, more sophisticated pool growth options is something I want longer term. We can also provide knobs that will let you drive the pool growth logic yourself if needed.
- solatic 3y ago
  Cold start time is the issue with most serverless runtimes. Your own mission statement states: "We want on-demand, granular elastic scale of specific parts of our app code." Doing that correctly is fundamentally a question of how long you need to wait for cold starts, because if you have a traffic spike, the spiked part of the traffic is simply not being served until the cold start period elapses. If you're running hot runners with no load, or if you have incoming load without runners (immediately) serving them, then you're not really delivering on your goal here. AWS EC2 has had autoscaling groups for more than a decade, and of course, a VM is essentially a more elaborate wrapper for any kind of application code you can write, and one with a longer cold-start time. > Under load you are rarely paying the cold start time because the pool is already hot. My spiky workloads beg to differ.
  - bo0tzz 3y ago
    Depending of course on the workload and request volume, I imagine you could apply a strategy where code is run locally while waiting for a remote node to start up, so you can still serve the requests on time?
    - solatic 3y ago
      No, because then you're dividing the resources allocated to the function among the existing run + the new run. If you over-allocate ahead of time to accommodate for this, you might as well just run ordinary VMs, which always have excess allocation locally; the core idea of scaling granularly is that you only allocate the resources you need for that single execution (paying a premium compared to a VM but less overall for spiky workloads since less overhead will be wasted).
  - conradfr 3y ago
    In a Elixir/Phoenix app I don't think this will be really used for web traffic and more for background/async jobs.

by gitgud 3y ago

chrismccord 3y ago

The FLAME.Pool discussed later in the post addresses this. Runners are pooled and remain configurable hot for whatever time you want before idling down. Under load you are rarely paying the cold start time because the pool is already hot. We are also adding more sophisticated pool growth techniques to the Elixir library next so you also avoid hitting an at capacity runner and cold starting one. For hot runners, the only overhead is the latency between the parent and child, which should be the same datacenter so 1ms or sub 1ms.

bo0tzz 3y ago
Currently the per-runner concurrency is limited by a fixed number. Have you thought about approaches that instead base this on resource usage, so that runners can be used optimally?
- chrismccord 3y ago
  Yes, more sophisticated pool growth options is something I want longer term. We can also provide knobs that will let you drive the pool growth logic yourself if needed.
solatic 3y ago
Cold start time is the issue with most serverless runtimes. Your own mission statement states: "We want on-demand, granular elastic scale of specific parts of our app code." Doing that correctly is fundamentally a question of how long you need to wait for cold starts, because if you have a traffic spike, the spiked part of the traffic is simply not being served until the cold start period elapses. If you're running hot runners with no load, or if you have incoming load without runners (immediately) serving them, then you're not really delivering on your goal here. AWS EC2 has had autoscaling groups for more than a decade, and of course, a VM is essentially a more elaborate wrapper for any kind of application code you can write, and one with a longer cold-start time. > Under load you are rarely paying the cold start time because the pool is already hot. My spiky workloads beg to differ.
- bo0tzz 3y ago
  Depending of course on the workload and request volume, I imagine you could apply a strategy where code is run locally while waiting for a remote node to start up, so you can still serve the requests on time?
  - solatic 3y ago
    No, because then you're dividing the resources allocated to the function among the existing run + the new run. If you over-allocate ahead of time to accommodate for this, you might as well just run ordinary VMs, which always have excess allocation locally; the core idea of scaling granularly is that you only allocate the resources you need for that single execution (paying a premium compared to a VM but less overall for spiky workloads since less overhead will be wasted).
- conradfr 3y ago
  In a Elixir/Phoenix app I don't think this will be really used for web traffic and more for background/async jobs.