6 ms·

> To reduce latency, a common approach is to keep a VM idle for 10 minutes or longer after a function invocation to allow for potential reuse [45]. In contrast,

by zokier 3y ago

> To reduce latency, a common approach is to keep a VM idle for 10 minutes or longer after a function invocation to allow for potential reuse [45]. In contrast, if a FaaS platform is optimized for hardware utilization and throughput, this waiting time should be reduced by a factor of 10 or more, because starting a VM consumes significantly fewer resources than having a VM idle for 10 minutes This seems somewhat surprising in such controlled environment. Why does idle function consume so many resources? If I think a normal Linux system, an idle process doesn't really consume much resources at all

mike_hearn 3y ago
RAM, disk. Functions are often written in high level JITd languages, they may need to load large reference datasets from disk, they may require a lot of code to be transferred over the network before they can begin running, and because nobody trusts the security of the Linux kernel you also have to pay the VM startup time. An idle process on Linux isn't much different: it consumes RAM, disk and a kernel. If you trust the software you're running enough you can amortize the kernel cost but the others remain.
- dig1 3y ago
  > and because nobody trusts the security of the Linux kernel Seriously? I imagine all the infra worldwide is run on FaaS and short-lived VMs, right?
  - jitl 3y ago
    What they mean is that if you trusted the kernel you’d run functions in containers instead of micro-VMs, since containers need fewer resources and better start/stop latency.
    - mike_hearn 3y ago
      Yes. Old-style pre AWS clouds (ISPs with an FTP+LAMP stack) worked by using UNIX user sharing features. It went out of fashion because Linux wasn't good enough at work isolation and because it was insecure - too many local root escalation vulns. So now clouds all run on custom hypervisors. The VM/hypervisor interface is smaller and easier to secure than the userspace/kernelspace interface. Strangers don't share kernels, they share hypervisors. Unfortunately FaaS platforms actually suffer twice, because Linux userland is too chaotic to use directly. So you have to boot a clean VM with Docker in it, then install a container into that VM so the user can send you software in the now-standard format, then start up the function. Booting a Docker container from a standard running Linux system already takes ~100msec if I recall correctly. But 100msec of added latency can actually reduce usage and hurt revenue on big sites, so it's not acceptable. And that's just the inner container. Then you have the cost of booting Linux, cost of downloading the container (docker format is highly unoptimized) etc. So all these costs add up and then the only way to solve them is to amortize them. Oracle Cloud is working on a thing called GraalOS which is intended to help address this. It works by letting apps share a Linux kernel and using a userspace "supervisor" that relies on CFI, MPKs and NaCL style binary analysis to prevent code from connecting to the kernel directly. Containers and VMs are no longer necessary in that model and loading an app is just a case of copying it down to a node and mmapping it, but it does mean you have to be able to compile your app for this alternative pseudo-operating system. If you work with Java then you use native-image that makes Java apps compile to native code and start up super fast, so it works. https://graal.cloud/graalos/ https://graal.cloud/graalos/
      - nmhancoc 3y ago
        WASM also tries to solve this, essentially borrowing the isolation features from browsers for cheap startup latency. Cloudflare workers is the classic example.
willvarfar 3y ago
Are we talking about VMs (e.g. K8s or something) or JVMs (or other equiv garbage-collected runtimes for other languages with hot-code-loading)? I'm imagining that sharing actual JVMs is a massive saving over each function getting its own VM to launch a JVM in?
FooBarWidget 3y ago
I think it means reserving resources. When resources are reserved, they cannot be used for anything else (even if you're not actually using that reservation) so one can also say that those resources are "consumed".
bagels 3y ago
Memory is used.
Shrezzing 3y ago
A server idling on Function A can't be called for Function B. 144 servers idling for 10 minutes each before expiring makes up a day of compute-time for 1 server. For every 144 expected idle servers, Meta needs to purchase and maintain an additional physical machine to keep up with healthy throughput on xfaas. Every 15 minutes, the 100k server network has 1,500,000 compute-minutes available. For an extreme example, every 15 minutes, 50k DB Cleanup processes run for 5 minutes, and then sit idle for 10 minutes, totalling 15 mins each. In this scenario, to satisfy 250,000 demanded compute-minutes (16.6%), 750,000 compute-minutes were supplied (50%). All else equal, to keep up healthy throughput on the rest of the network, Meta needs to purchase and maintain an excess 33,400 (50%-16.6%) physical servers to satisfy this DB Cleanup process. Reducing the idle-time to 1 minute. The 50k process runs for 5 minutes, and then sits idle for 1 minute, totalling 6 min each. In this scenario, to satisfy 250,000 demanded compute-minutes (16.6%), 300,000 compute-minutes were supplied (20%). All else equal, to keep up healthy throughput on the rest of the network, Meta only needs to purchase and maintain an excess 4,000 (20%-16%) physical servers. That's an extreme example, but hopefully it demonstrates why at Meta's scale, the energy requirements of the servers can be secondary when optimising for hardware. Since optomising for hardware can make a difference large enough to mothball a moderately sized datacenter.
- zokier 3y ago
  > A server idling on Function A can't be called for Function B why not?
  - Shrezzing 3y ago
    The server in this context is a "warmed" Function VM. The first time a VM instantiates, it goes through all of the setup required to run a function. That includes generic reusable stuff like Operating System setup, but also Function-specific things like JIT compilation of the Function software, library loading, language loading, all of which are specific to the Function being executed. The Function VM is hyper-optomised to that one Function's task. There's compute-time costs in changing its role from Function A to Function B. In the public cloud, the only cost you save on is starting up the OS, if both Functions happen to use the same OS. It's faster in most public FaaS infrastructure settings to just destroy a Function VM than it is to transfer a VM from Function A's configuration into Function B's configuration. Similarly, it's faster to just leave the VM running for a little while, just in case the Function is invoked again. When the paper discusses VMs idling, that's the scenario they describe, where servers (or more accurately software VMs inside of physical servers), are locked into a specific function for some number of minutes. These minutes are dead computing time. Meta, as a private cloud provider to themselves, can do some interesting things to reduce the cold-start of their Functions, meaning they can run more efficient clouds if they choose, by massively reducing the idle-time that's an industry standard in the public cloud.

by zokier 3y ago

mike_hearn 3y ago

RAM, disk. Functions are often written in high level JITd languages, they may need to load large reference datasets from disk, they may require a lot of code to be transferred over the network before they can begin running, and because nobody trusts the security of the Linux kernel you also have to pay the VM startup time. An idle process on Linux isn't much different: it consumes RAM, disk and a kernel. If you trust the software you're running enough you can amortize the kernel cost but the others remain.

dig1 3y ago
> and because nobody trusts the security of the Linux kernel Seriously? I imagine all the infra worldwide is run on FaaS and short-lived VMs, right?
- jitl 3y ago
  What they mean is that if you trusted the kernel you’d run functions in containers instead of micro-VMs, since containers need fewer resources and better start/stop latency.
  - mike_hearn 3y ago
    Yes. Old-style pre AWS clouds (ISPs with an FTP+LAMP stack) worked by using UNIX user sharing features. It went out of fashion because Linux wasn't good enough at work isolation and because it was insecure - too many local root escalation vulns. So now clouds all run on custom hypervisors. The VM/hypervisor interface is smaller and easier to secure than the userspace/kernelspace interface. Strangers don't share kernels, they share hypervisors. Unfortunately FaaS platforms actually suffer twice, because Linux userland is too chaotic to use directly. So you have to boot a clean VM with Docker in it, then install a container into that VM so the user can send you software in the now-standard format, then start up the function. Booting a Docker container from a standard running Linux system already takes ~100msec if I recall correctly. But 100msec of added latency can actually reduce usage and hurt revenue on big sites, so it's not acceptable. And that's just the inner container. Then you have the cost of booting Linux, cost of downloading the container (docker format is highly unoptimized) etc. So all these costs add up and then the only way to solve them is to amortize them. Oracle Cloud is working on a thing called GraalOS which is intended to help address this. It works by letting apps share a Linux kernel and using a userspace "supervisor" that relies on CFI, MPKs and NaCL style binary analysis to prevent code from connecting to the kernel directly. Containers and VMs are no longer necessary in that model and loading an app is just a case of copying it down to a node and mmapping it, but it does mean you have to be able to compile your app for this alternative pseudo-operating system. If you work with Java then you use native-image that makes Java apps compile to native code and start up super fast, so it works. https://graal.cloud/graalos/ https://graal.cloud/graalos/
    - nmhancoc 3y ago
      WASM also tries to solve this, essentially borrowing the isolation features from browsers for cheap startup latency. Cloudflare workers is the classic example.

willvarfar 3y ago

Are we talking about VMs (e.g. K8s or something) or JVMs (or other equiv garbage-collected runtimes for other languages with hot-code-loading)? I'm imagining that sharing actual JVMs is a massive saving over each function getting its own VM to launch a JVM in?

FooBarWidget 3y ago

I think it means reserving resources. When resources are reserved, they cannot be used for anything else (even if you're not actually using that reservation) so one can also say that those resources are "consumed".

bagels 3y ago

Memory is used.

Shrezzing 3y ago

A server idling on Function A can't be called for Function B. 144 servers idling for 10 minutes each before expiring makes up a day of compute-time for 1 server. For every 144 expected idle servers, Meta needs to purchase and maintain an additional physical machine to keep up with healthy throughput on xfaas. Every 15 minutes, the 100k server network has 1,500,000 compute-minutes available. For an extreme example, every 15 minutes, 50k DB Cleanup processes run for 5 minutes, and then sit idle for 10 minutes, totalling 15 mins each. In this scenario, to satisfy 250,000 demanded compute-minutes (16.6%), 750,000 compute-minutes were supplied (50%). All else equal, to keep up healthy throughput on the rest of the network, Meta needs to purchase and maintain an excess 33,400 (50%-16.6%) physical servers to satisfy this DB Cleanup process. Reducing the idle-time to 1 minute. The 50k process runs for 5 minutes, and then sits idle for 1 minute, totalling 6 min each. In this scenario, to satisfy 250,000 demanded compute-minutes (16.6%), 300,000 compute-minutes were supplied (20%). All else equal, to keep up healthy throughput on the rest of the network, Meta only needs to purchase and maintain an excess 4,000 (20%-16%) physical servers. That's an extreme example, but hopefully it demonstrates why at Meta's scale, the energy requirements of the servers can be secondary when optimising for hardware. Since optomising for hardware can make a difference large enough to mothball a moderately sized datacenter.

zokier 3y ago
> A server idling on Function A can't be called for Function B why not?
- Shrezzing 3y ago
  The server in this context is a "warmed" Function VM. The first time a VM instantiates, it goes through all of the setup required to run a function. That includes generic reusable stuff like Operating System setup, but also Function-specific things like JIT compilation of the Function software, library loading, language loading, all of which are specific to the Function being executed. The Function VM is hyper-optomised to that one Function's task. There's compute-time costs in changing its role from Function A to Function B. In the public cloud, the only cost you save on is starting up the OS, if both Functions happen to use the same OS. It's faster in most public FaaS infrastructure settings to just destroy a Function VM than it is to transfer a VM from Function A's configuration into Function B's configuration. Similarly, it's faster to just leave the VM running for a little while, just in case the Function is invoked again. When the paper discusses VMs idling, that's the scenario they describe, where servers (or more accurately software VMs inside of physical servers), are locked into a specific function for some number of minutes. These minutes are dead computing time. Meta, as a private cloud provider to themselves, can do some interesting things to reduce the cold-start of their Functions, meaning they can run more efficient clouds if they choose, by massively reducing the idle-time that's an industry standard in the public cloud.