3 ms·
Beware that such techniques slow down memory access for the newly running VM quite substantially. For example, a simple memset() call across a few gigabytes of
by londons_explore 4y ago
Beware that such techniques slow down memory access for the newly running VM quite substantially.
For example, a simple memset() call across a few gigabytes of RAM inside the VM might slow down a factor of 1000x after a VM clone like this.
Both 'parent' and 'child' VM's see the slowdowns.
Some types of garbage collector also see really substantial slowdowns.
It was a deal-breaker for my project where I did similar sorts of things.
- CompuIves 4y agoCould you share why you're seeing such a slowdown? I've mainly experienced a slowdown when loading pages for the first time, since new pages need to be page faulted into memory from disk first, but after those pages have been loaded into memory I don't experience any slowdown compared to fresh starts. That said, that's only based on how I've experienced it with the type of workloads that we run on the VMs.
- londons_explore 4y agoIt's the page fault+copy latency, together with some secondary effects from the page tables being updated (seems to briefly halt all cores). The actual copying of a page of RAM is almost free compared to the time spent in all the kernel code for a page fault. If your RAM is file backed, you end up spending lots of time in the filesystem code too - I used anonymous mappings which really helped there, and called clone() on the VM process to keep them shared. I suspect if you use huge pages you might see lots of the impact vanish, but obviously that has other downsides.
- CompuIves 4y agoRight, that makes sense. Once the memory page is in memory it should be fast though. We use shared mapping right now, and practically the pages stay in memory during the lifetime of a VM once they've been loaded, but we need to do more testing when there's more memory pressure. I've been looking at huge pages recently, I'm going to do some more testing with transparent huge pages today and see if it changes performance. Unfortunately we cannot use reserved huge pages because that doesn't work with shared mmap on say an XFS FS. Another idea is to make clones use the same memory base layer of their parent, then the pages are already prefaulted and it would deduplicate overall memory usage. Many things to discover still..
- intelVISA 4y agoCopy on write on a VMM is solid for a naive impl. but you need heavier stuff for prod I fear.