7 ms·

Remove PG_ZERO and zeroidle (page-zeroing) entirely

142 points by protomyth 10y ago

wolf550e 10y ago
EDIT: Wrong thread. Meant to post here: https://news.ycombinator.com/item?id=12227507 https://news.ycombinator.com/item?id=12227507
- tomjakubowski 10y ago
  That's "PG" as in "page" not as in "postgres". It's a Linux kernel patch.
  - daveloyall 10y ago
    No, it's a DragonFlyBSD kernel patch.
  - Sanddancer 10y ago
    Not Linux, DragonflyBSD.
daveloyall 10y ago
I don't get it. Edit: Ahh, now I get it. https://news.ycombinator.com/item?id=12228540 https://news.ycombinator.com/item?id=12228540
- protomyth 10y ago
  I thought it was interesting for: Remove the PG_ZERO flag and remove all page-zeroing optimizations, entirely. Aftering doing a substantial amount of testing, these optimizations, which existed all the way back to CSRG BSD, no longer provide any benefit on a modern system.
  - cgag 10y ago
    I think for many people without any context that explanation isn't quite enough. jdub's comment explains it quite well though.
    - protomyth 10y ago
      The joys of linking to the original source. Slashdot does have one interesting advantage over HN in that putting a large explanation, often with multiple links, is expected when submitting something. I probably could have found a blog or article with a fuller explanation, but then you get into accusations of submitting "blog spam". I figured this is HN and someone like jdub would come along and write a great explanation if the topic was worth consideration.
  - deleted 10y ago
    [deleted]
duaneb 10y ago
I wonder if this has issues with recent research showing reducing the temperature of memory allows reading it after a restart—swap computers to dodge the fault and read away.
tener 10y ago
Interesting, anyone knows if Linux does any of this?
- jdub 10y ago
  Yes, Linux (and Windows for that matter) does everything DragonflyBSD just removed.
  - bch 10y ago
    What does this mean? They do the old-dragonfly way, or the new-dragonfly way ?
    - jdub 10y ago
      The old way.
  - cpeterso 10y ago
    I wonder how long until someone submits a Linux patch to remove idle page-zeroing. :)
    - koenigdavidmj 10y ago
      Just be careful not to submit one patch per file, times 1500 files.
  - bboreham 10y ago
    Windows does more of it, in the sense the heap manager likes to give freed blocks back to the OS, whereas UNIX programs traditionally never gave memory back until program exit. I had a massive production outage a few years back involving this behaviour on Windows; symptom was 100% CPU usage in-kernel on exactly two threads and no other processes doing anything much. Still recall that as one of the most interesting bugs to track down.
    - viraptor 10y ago
jdub 10y ago
When a program needs memory, the operating system maps a physical page of memory (4KB in most cases) into its virtual memory space. That page might have been in use by a different program only moments before, so the new program could go poking around in it to find interesting stuff. Like SSH keys or passwords or whatever. So the operating system has a responsibility to tidy up pre-loved pages before giving them out. Once upon a time, filling 4KB with zeroes was a costly operation. It took a bunch of CPU cycles, and worse, would have trashed memory caches closer to the CPU. So operating systems tend to have queues of discarded pages, background threads to zero them (when nothing more important is happening), and queues of zeroed pages to hand out when a program wants more memory. This change is DragonFly BSD saying, "Fuck it, we'll do it live". It turns out that with the speed of modern CPUs and the way memory caches work today, they reckon it's faster to just zero the memory synchronously, when the program requests it. No more queues, no more background work, no more weird cache behaviour.
- vardump 10y ago
  > Once upon a time, filling 4KB with zeroes was a costly operation. It took a bunch of CPU cycles, and worse, would have trashed memory caches closer to the CPU. Even Intel 486 can do that in 50 microseconds. Without cache. Probably in a few microseconds assuming L1 cache. So I wonder if the assumption has ever been true.
  - jdub 10y ago
    I wonder if the acceptance of complexity in page zeroing was more about mapping and zeroing multiple pages… which makes me curious about DragonflyBSD's approach to that, as well as zeroing superpages.
  - gumby 10y ago
    > So I wonder if the assumption has ever been true. This fear went back to the PDP-11 in Unix and even predates Unix. It was true even when disks were really really slow, as CPUs had small or no cache and lacked multiple functional units, much less hardware multithreading. A big PDP-10 mainframe might still have been .7 VAX MIPS -- the first one I programmed's only "high speed" memory was its DTL registers -- core was literally core memory. You can get a feel for what a big deal this was in the fact that it felt radical and expensive for Stroustrup to decide that all new objects would be zeroed. Or you can see in the original posting that there was special memory handling support for special devices -- finally eliminated by this patch!
vardump 10y ago
The main takeaway from this: - Pre-zeroing a page only takes 80ns on a modern cpu. vm_fault overhead in general is ~at least 1 microscond. - Multiple synth and build tests show that active idle-time zeroing of pages actually reduces performance somewhat and incidental allocations of already-zerod pages (from page-table tear-downs) do not affect performance in any meaningful way. Not surprising. I've seen so much code with comments explaining how expensive copying or zeroing memory is (but not actually measuring it), that that's why "zero-copy" techniques are used. Ending up doing so much work (spending even microseconds) to "save" the cost that turns out to be 100 ns or less.
blinkingled 10y ago
DragonFlyBSD is slowly but steadily becoming more attractive as a Linux/FreeBSD alternative. 4.6 has good support for various newer Intel GPUs, hammer is stable and the SMP and Networking performance is stellar. I had a 2.8TB backup archive that I have repeatedly tried to dedup with ZFS and the realtime while-you-are-writing architecture of ZFS dedup just kills the performance so badly that I have never actually succeeded going over the full dataset. Past week I made a 4.6-rc2 based DragonFly VM, attached a 3TB disk with RDM, formatted it with HAMMER, started rsync of the backed up data and ran hammer dedup every 30 min - I am now at 1.9TB used after the whole thing was done and a final dedup ran - no noticeable speed drops and I got around 900GB back!
- rincebrain 10y ago
  Did you try something like bup? (Of course, depending on your use case, that might not work out so well.) Not that I have anything but respect for DFBSD, but I'd have suggested trying a userland archiving tool with dedup before spinning up a VM + FS in it.
  - blinkingled 10y ago
    Yeah I did try it on btrfs IIRC - but it's far too complicated (as opposed to hammer dedup where everything is included and it just works) and doesn't cater to my use case too well - I have multiple machines backing files up to a SMB share with lot of potential for duplicate data. I also need to be able to access those backed up files back over SMB. Just making a HAMMER mount point, running dedup on it via cron job at night and exporting it via Samba does everything I need.
    - rincebrain 10y ago
      Yeah, that's much nicer for your use case than bup. (I should point out that Windows apparently has similar after-the-fact dedup capabilities on NTFS in Server 2012/R2 and up [1], though I suspect you'll find DFBSD much easier to run with low overhead in a VM.) [1] - https://msdn.microsoft.com/en-us/library/hh769303%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396 https://msdn.microsoft.com/en-us/library/hh769303%28v=vs.85%...
milcron 10y ago
I always love net-negative commits.
- daveloyall 10y ago
  Ah yes, this one is a removal commit by a wide margin. 66 files changed, 184 insertions(+), 446 deletions(-)
- unixhero 10y ago
  So zen
  - milcron 10y ago
    The best code is no code! Related, one of my favorite programming stories: the 0-byte program http://peetm.com/blog/?p=55 http://peetm.com/blog/?p=55
Animats 10y ago
Fear of zeroing is related to fear of copying. One of the big arguments against message-passing systems is that there's extra copying. But today, copying is usually cheap, especially if the data was just created and is in cache. On the other hand, futzing with the MMU to avoid copying usually results in lots of cache flushes and is a lose unless you're remapping a huge memory area. This is a total reversal from the situation back when, for example, Mach was designed. Or, for that matter, Linux. It makes message passing microkernels faster than they were back then. It helps to do copying right. If a user process does something that causes the operating system to copy user-created data, the copy should take place on the same CPU, where the cache is current. These sorts of issues are why message passing and CPU dispatching have to be integrated to get good performance from a microkernel.
ryuuchin 10y ago
If I understand this change correctly it removes background zeroing of freed pages. What effect will this have on security? Grsecurity adds a feature which allows the linux kernel to sanitize freed pages[1]. I realize that the idle time freeing isn't the same as the immediate sanitation that grsecurity offers but I'm curious if this change has any effect on security in this manner. [1] https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Sanitize_all_freed_memory https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity...
- PDoyle 10y ago
  Nope. I draw your attention to this paragraph: > Zeroing the page at fault-time...

142 points by protomyth 10y ago

EDIT: Wrong thread. Meant to post here: https://news.ycombinator.com/item?id=12227507 https://news.ycombinator.com/item?id=12227507

tomjakubowski 10y ago
That's "PG" as in "page" not as in "postgres". It's a Linux kernel patch.
- daveloyall 10y ago
  No, it's a DragonFlyBSD kernel patch.
- Sanddancer 10y ago
  Not Linux, DragonflyBSD.

daveloyall 10y ago

I don't get it. Edit: Ahh, now I get it. https://news.ycombinator.com/item?id=12228540 https://news.ycombinator.com/item?id=12228540

protomyth 10y ago
I thought it was interesting for: Remove the PG_ZERO flag and remove all page-zeroing optimizations, entirely. Aftering doing a substantial amount of testing, these optimizations, which existed all the way back to CSRG BSD, no longer provide any benefit on a modern system.
- cgag 10y ago
  I think for many people without any context that explanation isn't quite enough. jdub's comment explains it quite well though.
  - protomyth 10y ago
    The joys of linking to the original source. Slashdot does have one interesting advantage over HN in that putting a large explanation, often with multiple links, is expected when submitting something. I probably could have found a blog or article with a fuller explanation, but then you get into accusations of submitting "blog spam". I figured this is HN and someone like jdub would come along and write a great explanation if the topic was worth consideration.
- deleted 10y ago
  [deleted]

duaneb 10y ago

I wonder if this has issues with recent research showing reducing the temperature of memory allows reading it after a restart—swap computers to dodge the fault and read away.

tener 10y ago

Interesting, anyone knows if Linux does any of this?

jdub 10y ago
Yes, Linux (and Windows for that matter) does everything DragonflyBSD just removed.
- bch 10y ago
  What does this mean? They do the old-dragonfly way, or the new-dragonfly way ?
  - jdub 10y ago
    The old way.
- cpeterso 10y ago
  I wonder how long until someone submits a Linux patch to remove idle page-zeroing. :)
  - koenigdavidmj 10y ago
    Just be careful not to submit one patch per file, times 1500 files.
- bboreham 10y ago
  Windows does more of it, in the sense the heap manager likes to give freed blocks back to the OS, whereas UNIX programs traditionally never gave memory back until program exit. I had a massive production outage a few years back involving this behaviour on Windows; symptom was 100% CPU usage in-kernel on exactly two threads and no other processes doing anything much. Still recall that as one of the most interesting bugs to track down.
  - viraptor 10y ago

jdub 10y ago

When a program needs memory, the operating system maps a physical page of memory (4KB in most cases) into its virtual memory space. That page might have been in use by a different program only moments before, so the new program could go poking around in it to find interesting stuff. Like SSH keys or passwords or whatever. So the operating system has a responsibility to tidy up pre-loved pages before giving them out. Once upon a time, filling 4KB with zeroes was a costly operation. It took a bunch of CPU cycles, and worse, would have trashed memory caches closer to the CPU. So operating systems tend to have queues of discarded pages, background threads to zero them (when nothing more important is happening), and queues of zeroed pages to hand out when a program wants more memory. This change is DragonFly BSD saying, "Fuck it, we'll do it live". It turns out that with the speed of modern CPUs and the way memory caches work today, they reckon it's faster to just zero the memory synchronously, when the program requests it. No more queues, no more background work, no more weird cache behaviour.

vardump 10y ago
> Once upon a time, filling 4KB with zeroes was a costly operation. It took a bunch of CPU cycles, and worse, would have trashed memory caches closer to the CPU. Even Intel 486 can do that in 50 microseconds. Without cache. Probably in a few microseconds assuming L1 cache. So I wonder if the assumption has ever been true.
- jdub 10y ago
  I wonder if the acceptance of complexity in page zeroing was more about mapping and zeroing multiple pages… which makes me curious about DragonflyBSD's approach to that, as well as zeroing superpages.
- gumby 10y ago
  > So I wonder if the assumption has ever been true. This fear went back to the PDP-11 in Unix and even predates Unix. It was true even when disks were really really slow, as CPUs had small or no cache and lacked multiple functional units, much less hardware multithreading. A big PDP-10 mainframe might still have been .7 VAX MIPS -- the first one I programmed's only "high speed" memory was its DTL registers -- core was literally core memory. You can get a feel for what a big deal this was in the fact that it felt radical and expensive for Stroustrup to decide that all new objects would be zeroed. Or you can see in the original posting that there was special memory handling support for special devices -- finally eliminated by this patch!

vardump 10y ago

The main takeaway from this: - Pre-zeroing a page only takes 80ns on a modern cpu. vm_fault overhead in general is ~at least 1 microscond. - Multiple synth and build tests show that active idle-time zeroing of pages actually reduces performance somewhat and incidental allocations of already-zerod pages (from page-table tear-downs) do not affect performance in any meaningful way. Not surprising. I've seen so much code with comments explaining how expensive copying or zeroing memory is (but not actually measuring it), that that's why "zero-copy" techniques are used. Ending up doing so much work (spending even microseconds) to "save" the cost that turns out to be 100 ns or less.

blinkingled 10y ago

DragonFlyBSD is slowly but steadily becoming more attractive as a Linux/FreeBSD alternative. 4.6 has good support for various newer Intel GPUs, hammer is stable and the SMP and Networking performance is stellar. I had a 2.8TB backup archive that I have repeatedly tried to dedup with ZFS and the realtime while-you-are-writing architecture of ZFS dedup just kills the performance so badly that I have never actually succeeded going over the full dataset. Past week I made a 4.6-rc2 based DragonFly VM, attached a 3TB disk with RDM, formatted it with HAMMER, started rsync of the backed up data and ran hammer dedup every 30 min - I am now at 1.9TB used after the whole thing was done and a final dedup ran - no noticeable speed drops and I got around 900GB back!

rincebrain 10y ago
Did you try something like bup? (Of course, depending on your use case, that might not work out so well.) Not that I have anything but respect for DFBSD, but I'd have suggested trying a userland archiving tool with dedup before spinning up a VM + FS in it.
- blinkingled 10y ago
  Yeah I did try it on btrfs IIRC - but it's far too complicated (as opposed to hammer dedup where everything is included and it just works) and doesn't cater to my use case too well - I have multiple machines backing files up to a SMB share with lot of potential for duplicate data. I also need to be able to access those backed up files back over SMB. Just making a HAMMER mount point, running dedup on it via cron job at night and exporting it via Samba does everything I need.
  - rincebrain 10y ago
    Yeah, that's much nicer for your use case than bup. (I should point out that Windows apparently has similar after-the-fact dedup capabilities on NTFS in Server 2012/R2 and up [1], though I suspect you'll find DFBSD much easier to run with low overhead in a VM.) [1] - https://msdn.microsoft.com/en-us/library/hh769303%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396 https://msdn.microsoft.com/en-us/library/hh769303%28v=vs.85%...

milcron 10y ago

I always love net-negative commits.

daveloyall 10y ago
Ah yes, this one is a removal commit by a wide margin. 66 files changed, 184 insertions(+), 446 deletions(-)
unixhero 10y ago
So zen
- milcron 10y ago
  The best code is no code! Related, one of my favorite programming stories: the 0-byte program http://peetm.com/blog/?p=55 http://peetm.com/blog/?p=55

Animats 10y ago

Fear of zeroing is related to fear of copying. One of the big arguments against message-passing systems is that there's extra copying. But today, copying is usually cheap, especially if the data was just created and is in cache. On the other hand, futzing with the MMU to avoid copying usually results in lots of cache flushes and is a lose unless you're remapping a huge memory area. This is a total reversal from the situation back when, for example, Mach was designed. Or, for that matter, Linux. It makes message passing microkernels faster than they were back then. It helps to do copying right. If a user process does something that causes the operating system to copy user-created data, the copy should take place on the same CPU, where the cache is current. These sorts of issues are why message passing and CPU dispatching have to be integrated to get good performance from a microkernel.

ryuuchin 10y ago

If I understand this change correctly it removes background zeroing of freed pages. What effect will this have on security? Grsecurity adds a feature which allows the linux kernel to sanitize freed pages[1]. I realize that the idle time freeing isn't the same as the immediate sanitation that grsecurity offers but I'm curious if this change has any effect on security in this manner. [1] https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Sanitize_all_freed_memory https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity...

PDoyle 10y ago
Nope. I draw your attention to this paragraph: > Zeroing the page at fault-time...