3 ms·

A server idling on Function A can't be called for Function B. 144 servers idling for 10 minutes each before expiring makes up a day of compute-time for 1 server

A server idling on Function A can't be called for Function B. 144 servers idling for 10 minutes each before expiring makes up a day of compute-time for 1 server. For every 144 expected idle servers, Meta needs to purchase and maintain an additional physical machine to keep up with healthy throughput on xfaas. Every 15 minutes, the 100k server network has 1,500,000 compute-minutes available. For an extreme example, every 15 minutes, 50k DB Cleanup processes run for 5 minutes, and then sit idle for 10 minutes, totalling 15 mins each. In this scenario, to satisfy 250,000 demanded compute-minutes (16.6%), 750,000 compute-minutes were supplied (50%). All else equal, to keep up healthy throughput on the rest of the network, Meta needs to purchase and maintain an excess 33,400 (50%-16.6%) physical servers to satisfy this DB Cleanup process. Reducing the idle-time to 1 minute. The 50k process runs for 5 minutes, and then sits idle for 1 minute, totalling 6 min each. In this scenario, to satisfy 250,000 demanded compute-minutes (16.6%), 300,000 compute-minutes were supplied (20%). All else equal, to keep up healthy throughput on the rest of the network, Meta only needs to purchase and maintain an excess 4,000 (20%-16%) physical servers. That's an extreme example, but hopefully it demonstrates why at Meta's scale, the energy requirements of the servers can be secondary when optimising for hardware. Since optomising for hardware can make a difference large enough to mothball a moderately sized datacenter.

zokier 3y ago
> A server idling on Function A can't be called for Function B why not?
- Shrezzing 3y ago
  The server in this context is a "warmed" Function VM. The first time a VM instantiates, it goes through all of the setup required to run a function. That includes generic reusable stuff like Operating System setup, but also Function-specific things like JIT compilation of the Function software, library loading, language loading, all of which are specific to the Function being executed. The Function VM is hyper-optomised to that one Function's task. There's compute-time costs in changing its role from Function A to Function B. In the public cloud, the only cost you save on is starting up the OS, if both Functions happen to use the same OS. It's faster in most public FaaS infrastructure settings to just destroy a Function VM than it is to transfer a VM from Function A's configuration into Function B's configuration. Similarly, it's faster to just leave the VM running for a little while, just in case the Function is invoked again. When the paper discusses VMs idling, that's the scenario they describe, where servers (or more accurately software VMs inside of physical servers), are locked into a specific function for some number of minutes. These minutes are dead computing time. Meta, as a private cloud provider to themselves, can do some interesting things to reduce the cold-start of their Functions, meaning they can run more efficient clouds if they choose, by massively reducing the idle-time that's an industry standard in the public cloud.
  - zokier 3y ago
    > There's compute-time costs in changing its role from Function A to Function B. Why do you need to change existing VM in any way? Having an idle VM for function A should not prevent the host from instantiating new VMs for function B. I still do not see where is the cost of an idle VM. > servers (or more accurately software VMs inside of physical servers), are locked into a specific function I think this needs to be more specific; what resources are exactly locked to a VM? Are they pinning VMs to specific CPUs or something that prevents the host from scheduling other tasks there? If so, why? Also, you sound very confident in your answers, do you have some additional sources you could point me to, or are you also basing all this on this one paper?
    - Amezarak 3y ago
      > Why do you need to change existing VM in any way? Having an idle VM for function A should not prevent the host from instantiating new VMs for function B. I still do not see where is the cost of an idle VM. Because the host can only run so many VMs. What's being described is host resource exhaustion from running idle VMs.
      - zokier 3y ago
        But that doesn't still explain the original statement > waiting time should be reduced by a factor of 10 or more, because starting a VM consumes significantly fewer resources than having a VM idle for 10 minutes To me its not obvious why you would want to have time-based eviction of VMs ever. Surely it would be more efficient to evict VMs only as response to some resource pressure? Basically I'm imagining keeping VMs in some LRU style structure where they could get kicked out when needed instead of just having 1min timer. But that is on abstract level, on concrete level its also not obvious at all what are the constraining resources that VMs consume? Especially in a sense that would be comparable to the resource consumption of starting a new VM. I realize that they are probably implying that they have some sort of static resource allocation for VMs meaning that idle VM consumes as much "resources" as active VM. But its not stated anywhere in the paper and it is weird to make claims(/recommendations) that hinge on such silent assumptions, especially if they are in no way universal. I feel this thing of improving utilization be reducing idleness is somewhat central part of the paper, so that's why I latched on this thing. I also realize that managing resources efficiently with VMs is bit more involved than traditional processes (=containers), but at the same time this is where Meta, running private cloud and practically owning the whole stack, could really do much more than public cloud ever could. They could rely on more VMs and host cooperating on resource allocation, and even have the possible language runtime (HHVM etc) cooperate here somehow. Indeed now that I think about it, sounds like interesting question how would you design FaaS-optimized combined VMM, VM, and language runtime stack. They touch on that with their JIT caches, but that seems one fairly narrow optimization, I'd imagine there is lot you could do here. The end-game might look something like unikernels, or maybe something completely different. (and I'm sorry if my comments have come off as combative, but this did stick out to me and I am just curious)

zokier 3y ago

> A server idling on Function A can't be called for Function B why not?

Shrezzing 3y ago
The server in this context is a "warmed" Function VM. The first time a VM instantiates, it goes through all of the setup required to run a function. That includes generic reusable stuff like Operating System setup, but also Function-specific things like JIT compilation of the Function software, library loading, language loading, all of which are specific to the Function being executed. The Function VM is hyper-optomised to that one Function's task. There's compute-time costs in changing its role from Function A to Function B. In the public cloud, the only cost you save on is starting up the OS, if both Functions happen to use the same OS. It's faster in most public FaaS infrastructure settings to just destroy a Function VM than it is to transfer a VM from Function A's configuration into Function B's configuration. Similarly, it's faster to just leave the VM running for a little while, just in case the Function is invoked again. When the paper discusses VMs idling, that's the scenario they describe, where servers (or more accurately software VMs inside of physical servers), are locked into a specific function for some number of minutes. These minutes are dead computing time. Meta, as a private cloud provider to themselves, can do some interesting things to reduce the cold-start of their Functions, meaning they can run more efficient clouds if they choose, by massively reducing the idle-time that's an industry standard in the public cloud.
- zokier 3y ago
  > There's compute-time costs in changing its role from Function A to Function B. Why do you need to change existing VM in any way? Having an idle VM for function A should not prevent the host from instantiating new VMs for function B. I still do not see where is the cost of an idle VM. > servers (or more accurately software VMs inside of physical servers), are locked into a specific function I think this needs to be more specific; what resources are exactly locked to a VM? Are they pinning VMs to specific CPUs or something that prevents the host from scheduling other tasks there? If so, why? Also, you sound very confident in your answers, do you have some additional sources you could point me to, or are you also basing all this on this one paper?
  - Amezarak 3y ago
    > Why do you need to change existing VM in any way? Having an idle VM for function A should not prevent the host from instantiating new VMs for function B. I still do not see where is the cost of an idle VM. Because the host can only run so many VMs. What's being described is host resource exhaustion from running idle VMs.
    - zokier 3y ago
      But that doesn't still explain the original statement > waiting time should be reduced by a factor of 10 or more, because starting a VM consumes significantly fewer resources than having a VM idle for 10 minutes To me its not obvious why you would want to have time-based eviction of VMs ever. Surely it would be more efficient to evict VMs only as response to some resource pressure? Basically I'm imagining keeping VMs in some LRU style structure where they could get kicked out when needed instead of just having 1min timer. But that is on abstract level, on concrete level its also not obvious at all what are the constraining resources that VMs consume? Especially in a sense that would be comparable to the resource consumption of starting a new VM. I realize that they are probably implying that they have some sort of static resource allocation for VMs meaning that idle VM consumes as much "resources" as active VM. But its not stated anywhere in the paper and it is weird to make claims(/recommendations) that hinge on such silent assumptions, especially if they are in no way universal. I feel this thing of improving utilization be reducing idleness is somewhat central part of the paper, so that's why I latched on this thing. I also realize that managing resources efficiently with VMs is bit more involved than traditional processes (=containers), but at the same time this is where Meta, running private cloud and practically owning the whole stack, could really do much more than public cloud ever could. They could rely on more VMs and host cooperating on resource allocation, and even have the possible language runtime (HHVM etc) cooperate here somehow. Indeed now that I think about it, sounds like interesting question how would you design FaaS-optimized combined VMM, VM, and language runtime stack. They touch on that with their JIT caches, but that seems one fairly narrow optimization, I'd imagine there is lot you could do here. The end-game might look something like unikernels, or maybe something completely different. (and I'm sorry if my comments have come off as combative, but this did stick out to me and I am just curious)