3 ms·

The kernel doesn't really have any tricks that userspace doesn't have to zero arrays quickly. In fact, it is somewhat hamstrung when zeroing since use of SIMD i

The kernel doesn't really have any tricks that userspace doesn't have to zero arrays quickly. In fact, it is somewhat hamstrung when zeroing since use of SIMD instructions in the kernel is discouraged and generally avoided. It usually ends up using `rep stosb` which is nice and fast on modern Intel (up to 32 bytes/cycle for AVX-supporting boxes), so that's not currently a problem, but it was slower in the past. Furthermore, this doesn't help you for repeatedly zeroing existing memory, which is probably the most important: are you going to mmap fresh pages every time you want zeros in there rather than just zeroing in userspace the memory you have?

toast0 7y ago
> The kernel doesn't really have any tricks that userspace doesn't have to zero arrays quickly. The kernel does have some tricks available --- it can't clean pages any faster than userspace, but it can keep a cache of clean pages, and keep the cache filled by cleaning free pages during idle, including before your program was launched. I'm sure that the kernel was able to do the work before your program was launched will look great in microbenchmarks, if used right, but if you need to do it repeatedly, mmunap/mmap syscall processing time is going to be a lot bigger than rep stosb, unless you have a giant block to zero out and the block is nicely aligned. Another comment in this general thread suggests that Linux doesn't do idle zeroing, but it's not the only kernel; FreeBSD has idle zeroing [1]. [1] https://www.freebsd.org/doc/en/articles/vm-design/prefault-optimizations.html https://www.freebsd.org/doc/en/articles/vm-design/prefault-o...
nwallin 7y ago
This isn't actually true. The kernel has two giant tricks up its sleeve when it comes to serving zero pages: the kernel gets to decide what happens when a page fault happens, and it gets to decide what to tell the mmu. At boot, the kernel allocates a physical page of RAM, zeroes it, and marks it read only. Some time later, a user process requests a zeroed page. The kernel finds a new address in virtual memory, and tells the mmu to point that address to the zeroed page. The user process reads from the page, and gets zeroes back, because that's what's in the zero page. And there's nothing wrong with reading a read only page. Then the user tries to write into the zero page, and a page fault happens, because the page is read only. The kernel allocates new physical RAM, zeroes it, and updates the mmu mapping. This doesn't appear to do any less work, but gigabytes of virtual memory can in 4kB of cache, because it's all the same page. So it's way faster. One of my favorite benchmarks of all time is creating an identity matrix by using malloc, memset(0) it, then assign ones to the diagonal, vs using calloc, (which as per its specification zeroes it) and assigning ones to the diagonal. Then comparing the run time of multiplying the two identity matrices by some other matrix. The calloc identity matrix is multiple orders of magnitude faster.
- BeeOnRope 7y ago
  Sure, I know how the zero page works - so for "sparse" scenarios like your diagonal matrix examples, it works great. You can of course also implement this trick in userspace, or at least without relying on the zero CoW thig on fault, e.g., by mapping /dev/zero. Sometimes the zero page thing comes out worse: if you first read from a zero page and then ultimately write to it, for every page, you take 2x the number of faults.

toast0 7y ago

> The kernel doesn't really have any tricks that userspace doesn't have to zero arrays quickly. The kernel does have some tricks available --- it can't clean pages any faster than userspace, but it can keep a cache of clean pages, and keep the cache filled by cleaning free pages during idle, including before your program was launched. I'm sure that the kernel was able to do the work before your program was launched will look great in microbenchmarks, if used right, but if you need to do it repeatedly, mmunap/mmap syscall processing time is going to be a lot bigger than rep stosb, unless you have a giant block to zero out and the block is nicely aligned. Another comment in this general thread suggests that Linux doesn't do idle zeroing, but it's not the only kernel; FreeBSD has idle zeroing [1]. [1] https://www.freebsd.org/doc/en/articles/vm-design/prefault-optimizations.html https://www.freebsd.org/doc/en/articles/vm-design/prefault-o...

nwallin 7y ago

This isn't actually true. The kernel has two giant tricks up its sleeve when it comes to serving zero pages: the kernel gets to decide what happens when a page fault happens, and it gets to decide what to tell the mmu. At boot, the kernel allocates a physical page of RAM, zeroes it, and marks it read only. Some time later, a user process requests a zeroed page. The kernel finds a new address in virtual memory, and tells the mmu to point that address to the zeroed page. The user process reads from the page, and gets zeroes back, because that's what's in the zero page. And there's nothing wrong with reading a read only page. Then the user tries to write into the zero page, and a page fault happens, because the page is read only. The kernel allocates new physical RAM, zeroes it, and updates the mmu mapping. This doesn't appear to do any less work, but gigabytes of virtual memory can in 4kB of cache, because it's all the same page. So it's way faster. One of my favorite benchmarks of all time is creating an identity matrix by using malloc, memset(0) it, then assign ones to the diagonal, vs using calloc, (which as per its specification zeroes it) and assigning ones to the diagonal. Then comparing the run time of multiplying the two identity matrices by some other matrix. The calloc identity matrix is multiple orders of magnitude faster.

BeeOnRope 7y ago
Sure, I know how the zero page works - so for "sparse" scenarios like your diagonal matrix examples, it works great. You can of course also implement this trick in userspace, or at least without relying on the zero CoW thig on fault, e.g., by mapping /dev/zero. Sometimes the zero page thing comes out worse: if you first read from a zero page and then ultimately write to it, for every page, you take 2x the number of faults.