4 ms·

AFAIK The kernel needs to perform calls to the hypervisor in order to copy the page table, but I'm not expert enough to provide you with details about this unfo

by antirez 12y ago

AFAIK The kernel needs to perform calls to the hypervisor in order to copy the page table, but I'm not expert enough to provide you with details about this unfortunately. It is for sure not inherently due to virtualization, since for example VMware does not have this issue.

willvarfar 12y ago
Me sees you've struggled with the problem for a while too: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 filed April 2012 and not even an ack.
- antirez 12y ago
  Yes, at some point I got a report about Xen 3.0 (If I remember correctly) fixing the issue, but I never see in the real world things improving much AFAIK. Here is a table that shows fork times with different environments, just to show how bad the thing is: Linux beefy VM on VMware 6.0GB: 12.8 milliseconds per GB. Linux running on physical machine (Unknown HW): 13.1 milliseconds per GB. Linux running on physical machine (Xeon @ 2.27Ghz): 9 milliseconds per GB. Linux VM on 6sync (KVM): 23.3 millisecond per GB. Linux VM on EC2 (Xen): 239.3 milliseconds per GB. Linux VM on Linode (Xen): 424 milliseconds per GB. Around 30 times slower than bare metal, and I'm talking about old physical servers with slow memory compared to today's. EDIT: Make sure to read this -> https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJNvtaiPEI https://redislabs.com/blog/testing-fork-time-on-awsxen-infra...
  - daviesliu 12y ago
    Have you tried to use huge page to reduce the time of fork()? There also a similar problem for master of MFS, it will be great if you have some experience in it. EDIT: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
    - DrJosiah 12y ago
      At least one user of Redis tried to enable huge pages and reported that there were issues: https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9QowcJ https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9...
wtallis 12y ago
That sounds like a PV vs. HVM issue. Using hardware virtualization extensions to handle virtual memory is almost always faster than paravirtualization these days, which is why Xen introduced PVH mode in 4.4.
- derefr 12y ago
  Yep. And EC2's "HVM"-type instances are now actually PVHVM, not pure HVM. Since this change, there has been absolutely no reason to use anything other than HVM AMIs, and pure paravirtual instances can basically be considered a deprecated feature in EC2. New EC2 instance classes (e.g. t2) don't even support PV. Basically, PV instances are just there to support current customers who are relying on already-built PV AMIs and have too much inertial to be nudged into switching over.

by antirez 12y ago

willvarfar 12y ago

Me sees you've struggled with the problem for a while too: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1815 filed April 2012 and not even an ack.

antirez 12y ago
Yes, at some point I got a report about Xen 3.0 (If I remember correctly) fixing the issue, but I never see in the real world things improving much AFAIK. Here is a table that shows fork times with different environments, just to show how bad the thing is: Linux beefy VM on VMware 6.0GB: 12.8 milliseconds per GB. Linux running on physical machine (Unknown HW): 13.1 milliseconds per GB. Linux running on physical machine (Xeon @ 2.27Ghz): 9 milliseconds per GB. Linux VM on 6sync (KVM): 23.3 millisecond per GB. Linux VM on EC2 (Xen): 239.3 milliseconds per GB. Linux VM on Linode (Xen): 424 milliseconds per GB. Around 30 times slower than bare metal, and I'm talking about old physical servers with slow memory compared to today's. EDIT: Make sure to read this -> https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJNvtaiPEI https://redislabs.com/blog/testing-fork-time-on-awsxen-infra...
- daviesliu 12y ago
  Have you tried to use huge page to reduce the time of fork()? There also a similar problem for master of MFS, it will be great if you have some experience in it. EDIT: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
  - DrJosiah 12y ago
    At least one user of Redis tried to enable huge pages and reported that there were issues: https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9QowcJ https://groups.google.com/d/msg/redis-db/3VCDYKXhDdI/rIaPnK9...

wtallis 12y ago

That sounds like a PV vs. HVM issue. Using hardware virtualization extensions to handle virtual memory is almost always faster than paravirtualization these days, which is why Xen introduced PVH mode in 4.4.

derefr 12y ago
Yep. And EC2's "HVM"-type instances are now actually PVHVM, not pure HVM. Since this change, there has been absolutely no reason to use anything other than HVM AMIs, and pure paravirtual instances can basically be considered a deprecated feature in EC2. New EC2 instance classes (e.g. t2) don't even support PV. Basically, PV instances are just there to support current customers who are relying on already-built PV AMIs and have too much inertial to be nudged into switching over.