5 ms·

I’ve always wondered how much CPU time and memory bandwidth is taken up by the OS zeroing out pages before handing them out, as well as programs and libraries c

I’ve always wondered how much CPU time and memory bandwidth is taken up by the OS zeroing out pages before handing them out, as well as programs and libraries clearing chunks of memory. I guess it’s enough that I’m surprised that there’s no hardware support for the memory system to support a way to handle it by itself on command, without taking up bus bandwidth or CPU cycles. Kind of like old-fashioned REP STOS but handled off-chip, as it were. [Added:] Concerning various instructions for clearing whole cache lines in one go, you still end up with lots of dirty cache lines that have to be sent to L1, L2, ..., RAM (not to mention the stuff that was previously in those cache lines), so there’s still lots of bus bandwidth being consumed.

Doxin 7y ago
I'm sure there's some DMA going on there. The OS could simply keep a reserved zeroed out page and tell the DMA hardware to copy it wherever when it needs to zero out a page. I'd be rather surprised if tricks like that don't already happen. I'd be doubly surprised if modern DMA hardware doesn't support fancier tricks, if writing a repeated byte pattern is supported you wouldn't even need to reserve a whole page of zeroes, a single byte would do.
- vardump 7y ago
  DMA implies going to DRAM or in some cases L3. It's going to be way slower.
gpderetta 7y ago
One issue is that the memory controller is in the cpu itself (what goes on the wire are the low level line control signals). Also for small ranges, the memory actually being zeroed would be in cache anyway and anything touching the cache would interfere with the cpu, so it is hard to do better than rep stos.
- BeeOnRope 7y ago
  Yeah, this. Unless you are zeroing a really big chunk of memory, you expect that it might already be in the cache and you almost always want it in the cache after, so pushing it down to the memory is often counter-productive in this sense: you want it close to the core. Zeroing memory does take up a non-negligible amount of time some some workflows, especially those that make sparse use of the memory (e.g., don't end up writing all the memory in the end, think a large-radix histogram where many counters stay zero, or allocations that are larger than needed and don't end up getting used, etc). That said, zeroing can be pretty fast as this post shows: around 30 bytes per cycle, which on a 4 GHz CPU is 120 GB/s. AVX-512 doubles that to ~60 bytes/cycle.
gioele 7y ago
> I guess it’s enough that I’m surprised that there’s no hardware support for the memory system to support a way to handle it by itself on command, without taking up bus bandwidth or CPU cycles. Kind of like old-fashioned REP STOS but handled off-chip, as it were. Most ISAs provide a "zero this cache line" instruction: https://en.wikipedia.org/wiki/Cache_control_instruction#Data_cache_block_allocate_zero https://en.wikipedia.org/wiki/Cache_control_instruction#Data... I remember using `dcbz` (Data Cache Block Set to Zero) on PowerPC to quickly zero-out big chunks of memory.
pm215 7y ago
Not as far out as the memory system, but Arm provides 'DC ZVA' (https://developer.arm.com/docs/ddi0595/b/aarch64-system-instructions/dc-zva https://developer.arm.com/docs/ddi0595/b/aarch64-system-inst...) which clears a full cache line to zero at once, which is more efficient than a pile of explicit stores. (You'll see from the description that it correctly handles the case of a watchpoint being set on the memory to be cleared, or an attempt to clear non-writeable memory -- getting those cases right would I think be a lot harder to do outside the CPU.)
pingyong 7y ago
Zeroing memory is probably already really fast compared to other things the system is doing. Something that happened to me a couple years ago: I was writing a benchmark that needed a lot of memory. I just used a vector<char> for that. Of course, the vector zeroes the memory, so I thought hey, I don't actually need it to be zeroed, so why not just use unique_ptr<char[]>? Well it turns out (and most people here probably know what's coming next) that when you reserve memory, you don't actually get any physical pages. They get reserved as you access them. And since apparently the access pattern was worse (or something to that effect), the benchmark ended up running way slower than it did without explicit zeroing, even if I put the zeroing into the benchmark which it hadn't been previously. It turns out compared to reserving memory, zeroing was essentially inconsequential. So I'm not too optimistic that there are a lot of gains to be had here.
- nwallin 7y ago
  That's not actually what's happening. When you allocate memory that needs to be initialized to zeroes, it doesn't actually allocate memory and zero it. Nor does it allocate memory without mapping the pages, and when a page fault occurs, allocate and zero it. Here's what actually happens. Upon boot up, the OS allocates a page, zeros it, and marks it read only. When pages are requested in the future that need to be zero, it assigns an address and maps the zero page to it. This zero page could be mapped to hundreds, thousands of pages. If you try to read from the page, it happily tells you it contains zeroes. So you still haven't allocated new physical pages to that address, despite the fact that you're actively using it. Only when you try to write to it does a page fault happen. (because you're trying to write to a read only page) Then it allocates a new page of physical RAM, zeroes it, and gives you a real physical page to use. It's clever hacks and abstractions all the down.
  - nwmcsween 7y ago
    You're mixing up zeroing here, virtual memory is backed by physical memory and that is what is getting zeroed, in fact Linux recently got zeroing on free or malloc as a security feature.
_Nax 7y ago
> I’ve always wondered how much CPU time and memory bandwidth is taken up by the OS zeroing out pages before handing them out Not much, since on any modern system (that supports basic memory management in hardware), the OS have a single, special page full of zeroes and configured to trap on write that it maps every time a new page is requested. Only when a write occurs (and is trapped) does the OS substitute an actual page. It's also why on some of these systems, calloc is faster than malloc + memset : calloc may be special-cased not to zero anything on the userland side if it allocates a brand new set of pages, while the memset will always trigger the CoW (and write useless zeroes as well). Granted, the OS still needs to write zeroes a lot to implement the actual CoW operation, but it occurs less often than you would think.

Doxin 7y ago

I'm sure there's some DMA going on there. The OS could simply keep a reserved zeroed out page and tell the DMA hardware to copy it wherever when it needs to zero out a page. I'd be rather surprised if tricks like that don't already happen. I'd be doubly surprised if modern DMA hardware doesn't support fancier tricks, if writing a repeated byte pattern is supported you wouldn't even need to reserve a whole page of zeroes, a single byte would do.

vardump 7y ago
DMA implies going to DRAM or in some cases L3. It's going to be way slower.

gpderetta 7y ago

One issue is that the memory controller is in the cpu itself (what goes on the wire are the low level line control signals). Also for small ranges, the memory actually being zeroed would be in cache anyway and anything touching the cache would interfere with the cpu, so it is hard to do better than rep stos.

BeeOnRope 7y ago
Yeah, this. Unless you are zeroing a really big chunk of memory, you expect that it might already be in the cache and you almost always want it in the cache after, so pushing it down to the memory is often counter-productive in this sense: you want it close to the core. Zeroing memory does take up a non-negligible amount of time some some workflows, especially those that make sparse use of the memory (e.g., don't end up writing all the memory in the end, think a large-radix histogram where many counters stay zero, or allocations that are larger than needed and don't end up getting used, etc). That said, zeroing can be pretty fast as this post shows: around 30 bytes per cycle, which on a 4 GHz CPU is 120 GB/s. AVX-512 doubles that to ~60 bytes/cycle.

gioele 7y ago

> I guess it’s enough that I’m surprised that there’s no hardware support for the memory system to support a way to handle it by itself on command, without taking up bus bandwidth or CPU cycles. Kind of like old-fashioned REP STOS but handled off-chip, as it were. Most ISAs provide a "zero this cache line" instruction: https://en.wikipedia.org/wiki/Cache_control_instruction#Data_cache_block_allocate_zero https://en.wikipedia.org/wiki/Cache_control_instruction#Data... I remember using `dcbz` (Data Cache Block Set to Zero) on PowerPC to quickly zero-out big chunks of memory.

pm215 7y ago

Not as far out as the memory system, but Arm provides 'DC ZVA' (https://developer.arm.com/docs/ddi0595/b/aarch64-system-instructions/dc-zva https://developer.arm.com/docs/ddi0595/b/aarch64-system-inst...) which clears a full cache line to zero at once, which is more efficient than a pile of explicit stores. (You'll see from the description that it correctly handles the case of a watchpoint being set on the memory to be cleared, or an attempt to clear non-writeable memory -- getting those cases right would I think be a lot harder to do outside the CPU.)

pingyong 7y ago

Zeroing memory is probably already really fast compared to other things the system is doing. Something that happened to me a couple years ago: I was writing a benchmark that needed a lot of memory. I just used a vector<char> for that. Of course, the vector zeroes the memory, so I thought hey, I don't actually need it to be zeroed, so why not just use unique_ptr<char[]>? Well it turns out (and most people here probably know what's coming next) that when you reserve memory, you don't actually get any physical pages. They get reserved as you access them. And since apparently the access pattern was worse (or something to that effect), the benchmark ended up running way slower than it did without explicit zeroing, even if I put the zeroing into the benchmark which it hadn't been previously. It turns out compared to reserving memory, zeroing was essentially inconsequential. So I'm not too optimistic that there are a lot of gains to be had here.

nwallin 7y ago
That's not actually what's happening. When you allocate memory that needs to be initialized to zeroes, it doesn't actually allocate memory and zero it. Nor does it allocate memory without mapping the pages, and when a page fault occurs, allocate and zero it. Here's what actually happens. Upon boot up, the OS allocates a page, zeros it, and marks it read only. When pages are requested in the future that need to be zero, it assigns an address and maps the zero page to it. This zero page could be mapped to hundreds, thousands of pages. If you try to read from the page, it happily tells you it contains zeroes. So you still haven't allocated new physical pages to that address, despite the fact that you're actively using it. Only when you try to write to it does a page fault happen. (because you're trying to write to a read only page) Then it allocates a new page of physical RAM, zeroes it, and gives you a real physical page to use. It's clever hacks and abstractions all the down.
- nwmcsween 7y ago
  You're mixing up zeroing here, virtual memory is backed by physical memory and that is what is getting zeroed, in fact Linux recently got zeroing on free or malloc as a security feature.

_Nax 7y ago

> I’ve always wondered how much CPU time and memory bandwidth is taken up by the OS zeroing out pages before handing them out Not much, since on any modern system (that supports basic memory management in hardware), the OS have a single, special page full of zeroes and configured to trap on write that it maps every time a new page is requested. Only when a write occurs (and is trapped) does the OS substitute an actual page. It's also why on some of these systems, calloc is faster than malloc + memset : calloc may be special-cased not to zero anything on the userland side if it allocates a brand new set of pages, while the memset will always trigger the CoW (and write useless zeroes as well). Granted, the OS still needs to write zeroes a lot to implement the actual CoW operation, but it occurs less often than you would think.