5 ms·

Why is forking on xen slow? Most google hits for "xen forking slow" seem to point to some discussions about redis, but I guess other software would suffer from

Why is forking on xen slow? Most google hits for "xen forking slow" seem to point to some discussions about redis, but I guess other software would suffer from that too.

antirez 12y ago
AFAIK The kernel needs to perform calls to the hypervisor in order to copy the page table, but I'm not expert enough to provide you with details about this unfortunately. It is for sure not inherently due to virtualization, since for example VMware does not have this issue.
- willvarfar 12y ago
  Me sees you've struggled with the problem for a while too: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 filed April 2012 and not even an ack.
  - antirez 12y ago
    Yes, at some point I got a report about Xen 3.0 (If I remember correctly) fixing the issue, but I never see in the real world things improving much AFAIK. Here is a table that shows fork times with different environments, just to show how bad the thing is: Linux beefy VM on VMware 6.0GB: 12.8 milliseconds per GB. Linux running on physical machine (Unknown HW): 13.1 milliseconds per GB. Linux running on physical machine (Xeon @ 2.27Ghz): 9 milliseconds per GB. Linux VM on 6sync (KVM): 23.3 millisecond per GB. Linux VM on EC2 (Xen): 239.3 milliseconds per GB. Linux VM on Linode (Xen): 424 milliseconds per GB. Around 30 times slower than bare metal, and I'm talking about old physical servers with slow memory compared to today's. EDIT: Make sure to read this -> https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJNvtaiPEI https://redislabs.com/blog/testing-fork-time-on-awsxen-infra...
    - daviesliu 12y ago
      Have you tried to use huge page to reduce the time of fork()? There also a similar problem for master of MFS, it will be great if you have some experience in it. EDIT: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
      - DrJosiah 12y ago
        At least one user of Redis tried to enable huge pages and reported that there were issues: https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9QowcJ https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9...
- wtallis 12y ago
  That sounds like a PV vs. HVM issue. Using hardware virtualization extensions to handle virtual memory is almost always faster than paravirtualization these days, which is why Xen introduced PVH mode in 4.4.
  - derefr 12y ago
    Yep. And EC2's "HVM"-type instances are now actually PVHVM, not pure HVM. Since this change, there has been absolutely no reason to use anything other than HVM AMIs, and pure paravirtual instances can basically be considered a deprecated feature in EC2. New EC2 instance classes (e.g. t2) don't even support PV. Basically, PV instances are just there to support current customers who are relying on already-built PV AMIs and have too much inertial to be nudged into switching over.
r4um 12y ago
A thread on xen-devel, similar discussion. http://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html http://lists.xen.org/archives/html/xen-devel/2012-07/msg0008...
pja 12y ago
This mailing list post: http://lists.xen.org/archives/html/xen-devel/2012-06/msg01021.html http://lists.xen.org/archives/html/xen-devel/2012-06/msg0102... suggests that it's page table validation time, and it can be avoided if you use a PVHVM guest. Has anyone checked whether these redis problems apply only to PV guest images on EC2?
nknighthb 12y ago
It's important to remember two things: First, in a more general way, fork() is kind of slow just by virtue of being a system call and involving several steps, so applications try to avoid making lots of calls to fork (e.g. webservers long ago stopped doing the naïve fork-for-every-request model). Redis uses fork in a way nobody else seems to. Applications using tens of gigabytes of RAM -- databases, media editing, etc. -- just don't usually fork except during a startup process. Redis uses fork for persistence. It's clever, and it works great in general, but it's weird! Unique, even. It seems much more likely to hit a fork() weakness in a noticeable way than almost any other application out there.
- antirez 12y ago
  Totally true. I always think that eventually some kernel change may totally break how Redis works and I'll have to resort to implementing the same stuff in user space. A simpler parallel path is to use Redis Cluster / Sentinel and provide good safety for when persistence is disabled, and achieve data safety via redundancy in replicas. However even in such a scenario, it will be kind of handy that Redis can persist so that restarts performed with SHUTDOWN (for example for upgrades) will still retain the dataset when needed.
- JoeAltmaier 12y ago
  Fork has always been a mutant feature. Split the entire process into twins? So that the caller can immediately exec() and discard the tediously-cloned twin and become a different program? Its the most egregiously inefficient feature ever to grace an operating system. Page tables, stacks, heap allocations, file handles - all cloned and then discarded.
  - antirez 12y ago
    In this Regard, Redis is the first system using it the proper way I guess ;-) I mean, all this copying is not discarded but used to create a point-in-time snapshot.
    - mcguire 12y ago

Why is forking on xen slow? Most google hits for "xen forking slow" seem to point to some discussions about redis, but I guess other software would suffer from that too.

antirez 12y ago

AFAIK The kernel needs to perform calls to the hypervisor in order to copy the page table, but I'm not expert enough to provide you with details about this unfortunately. It is for sure not inherently due to virtualization, since for example VMware does not have this issue.

willvarfar 12y ago
Me sees you've struggled with the problem for a while too: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 filed April 2012 and not even an ack.
- antirez 12y ago
  Yes, at some point I got a report about Xen 3.0 (If I remember correctly) fixing the issue, but I never see in the real world things improving much AFAIK. Here is a table that shows fork times with different environments, just to show how bad the thing is: Linux beefy VM on VMware 6.0GB: 12.8 milliseconds per GB. Linux running on physical machine (Unknown HW): 13.1 milliseconds per GB. Linux running on physical machine (Xeon @ 2.27Ghz): 9 milliseconds per GB. Linux VM on 6sync (KVM): 23.3 millisecond per GB. Linux VM on EC2 (Xen): 239.3 milliseconds per GB. Linux VM on Linode (Xen): 424 milliseconds per GB. Around 30 times slower than bare metal, and I'm talking about old physical servers with slow memory compared to today's. EDIT: Make sure to read this -> https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJNvtaiPEI https://redislabs.com/blog/testing-fork-time-on-awsxen-infra...
  - daviesliu 12y ago
    Have you tried to use huge page to reduce the time of fork()? There also a similar problem for master of MFS, it will be great if you have some experience in it. EDIT: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
    - DrJosiah 12y ago
      At least one user of Redis tried to enable huge pages and reported that there were issues: https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9QowcJ https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9...
wtallis 12y ago
That sounds like a PV vs. HVM issue. Using hardware virtualization extensions to handle virtual memory is almost always faster than paravirtualization these days, which is why Xen introduced PVH mode in 4.4.
- derefr 12y ago
  Yep. And EC2's "HVM"-type instances are now actually PVHVM, not pure HVM. Since this change, there has been absolutely no reason to use anything other than HVM AMIs, and pure paravirtual instances can basically be considered a deprecated feature in EC2. New EC2 instance classes (e.g. t2) don't even support PV. Basically, PV instances are just there to support current customers who are relying on already-built PV AMIs and have too much inertial to be nudged into switching over.

r4um 12y ago

A thread on xen-devel, similar discussion. http://lists.xen.org/archives/html/xen-devel/2012-07/msg00085.html http://lists.xen.org/archives/html/xen-devel/2012-07/msg0008...

pja 12y ago

This mailing list post: http://lists.xen.org/archives/html/xen-devel/2012-06/msg01021.html http://lists.xen.org/archives/html/xen-devel/2012-06/msg0102... suggests that it's page table validation time, and it can be avoided if you use a PVHVM guest. Has anyone checked whether these redis problems apply only to PV guest images on EC2?

nknighthb 12y ago

It's important to remember two things: First, in a more general way, fork() is kind of slow just by virtue of being a system call and involving several steps, so applications try to avoid making lots of calls to fork (e.g. webservers long ago stopped doing the naïve fork-for-every-request model). Redis uses fork in a way nobody else seems to. Applications using tens of gigabytes of RAM -- databases, media editing, etc. -- just don't usually fork except during a startup process. Redis uses fork for persistence. It's clever, and it works great in general, but it's weird! Unique, even. It seems much more likely to hit a fork() weakness in a noticeable way than almost any other application out there.

antirez 12y ago
Totally true. I always think that eventually some kernel change may totally break how Redis works and I'll have to resort to implementing the same stuff in user space. A simpler parallel path is to use Redis Cluster / Sentinel and provide good safety for when persistence is disabled, and achieve data safety via redundancy in replicas. However even in such a scenario, it will be kind of handy that Redis can persist so that restarts performed with SHUTDOWN (for example for upgrades) will still retain the dataset when needed.
JoeAltmaier 12y ago
Fork has always been a mutant feature. Split the entire process into twins? So that the caller can immediately exec() and discard the tediously-cloned twin and become a different program? Its the most egregiously inefficient feature ever to grace an operating system. Page tables, stacks, heap allocations, file handles - all cloned and then discarded.
- antirez 12y ago
  In this Regard, Redis is the first system using it the proper way I guess ;-) I mean, all this copying is not discarded but used to create a point-in-time snapshot.
  - mcguire 12y ago