4 ms·

I used to use C a lot, but haven't in a while, so question: isn't there a sub-allocator involved also that could affect the condition of the memory allocated w

I used to use C a lot, but haven't in a while, so question: isn't there a sub-allocator involved also that could affect the condition of the memory allocated without an explicit zeroing ? Is the memory always allocated directly from the OS ? That seems like it could be slow. Thanks.

qb45 11y ago
Of course there is, the parent is clueless and spreads misinformation. Heap allocation can easily contain data that have been previously freed in the same process and hence calloc has to perform some extra work to wipe them.
- masklinn 11y ago
  > hence calloc has to perform some extra work to wipe them. IIRC FreeBSD will zero freed pages in the background when it has idle time, and calloc can draw from that pool of zeroed pages "for free".
  - DSMan195276 11y ago
    Ehh, that doesn't apply to what he's talking about though. That only works when the pages are actually freed. Since actually asking the kernel for more memory is expensive, most malloc/calloc implementations will hold-on to some pages since it assumes you're going to allocate more. Since it reuses this memory without ever giving it back to the kernel, the kernel won't zero it for us.
    - masklinn 11y ago
      > Ehh, that doesn't apply to what he's talking about though. Of course it does. > That only works when the pages are actually freed. Since actually asking the kernel for more memory is expensive, most malloc/calloc implementations will hold-on to some pages since it assumes you're going to allocate more. Since it reuses this memory without ever giving it back to the kernel, the kernel won't zero it for us. It's what the standard system malloc does, possibly in cooperation with the kernel but not necessarily so (why would it need the kernel involved after all?) Just as OpenBSD's malloc framework can do way more than trivially request memory from the kernel: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/malloc.conf.5?query=malloc%2econf http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/.... In fact, since OpenBSD 5.6 it'll junk small allocations on freeing (fill them with 0xdf) by default. And as usual that uncovered broken code… > When we started junking memory by default after free, the postgres ruby gem stopped working.
      - DSMan195276 11y ago
        Would you mind providing some more information on this? I googled it but I couldn't find any documentation on what you describing. To be clear though, the parent commenter (and I) are talking about the pages that malloc keeps around in the processes memory to be reused (And thus are never released back to the OS). Are you saying FreeBSD/OpenBSD/others have a system to tell the kernel when a user process has pages it plans to zero, and then a system for the kernel to notify the process when/if it does? That would be pretty interesting to see, but I've never heard of that being a thing.
        masklinn 11y ago
        > Would you mind providing some more information on this? I googled it but I couldn't find any documentation on what you describing. I was talking about https://www.freebsd.org/doc/en/articles/vm-design/prefault-optimizations.html https://www.freebsd.org/doc/en/articles/vm-design/prefault-o... but after actually re-checking the docs rather than going from bitrotted memory pre-zeroing only happens for the initial zero-fill fault, so the original calloc is "free" (if there are zeroed pages in the queue) but freeing then re-calloc'ing will need to zero the chunk (unless the allocator decides to go see if the kernel has pre-zeroed space instead I guess). My bad. > To be clear though, the parent commenter (and I) are talking about the pages that malloc keeps around in the processes memory to be reused (And thus are never released back to the OS). Yeah so was I, but I was misremembering stuff. > Are you saying FreeBSD/OpenBSD/others have a system to tell the kernel when a user process has pages it plans to zero, and then a system for the kernel to notify the process when/if it does? Turns out no.

qb45 11y ago

Of course there is, the parent is clueless and spreads misinformation. Heap allocation can easily contain data that have been previously freed in the same process and hence calloc has to perform some extra work to wipe them.

masklinn 11y ago
> hence calloc has to perform some extra work to wipe them. IIRC FreeBSD will zero freed pages in the background when it has idle time, and calloc can draw from that pool of zeroed pages "for free".
- DSMan195276 11y ago
  Ehh, that doesn't apply to what he's talking about though. That only works when the pages are actually freed. Since actually asking the kernel for more memory is expensive, most malloc/calloc implementations will hold-on to some pages since it assumes you're going to allocate more. Since it reuses this memory without ever giving it back to the kernel, the kernel won't zero it for us.
  - masklinn 11y ago
    > Ehh, that doesn't apply to what he's talking about though. Of course it does. > That only works when the pages are actually freed. Since actually asking the kernel for more memory is expensive, most malloc/calloc implementations will hold-on to some pages since it assumes you're going to allocate more. Since it reuses this memory without ever giving it back to the kernel, the kernel won't zero it for us. It's what the standard system malloc does, possibly in cooperation with the kernel but not necessarily so (why would it need the kernel involved after all?) Just as OpenBSD's malloc framework can do way more than trivially request memory from the kernel: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/malloc.conf.5?query=malloc%2econf http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/.... In fact, since OpenBSD 5.6 it'll junk small allocations on freeing (fill them with 0xdf) by default. And as usual that uncovered broken code… > When we started junking memory by default after free, the postgres ruby gem stopped working.
    - DSMan195276 11y ago
      Would you mind providing some more information on this? I googled it but I couldn't find any documentation on what you describing. To be clear though, the parent commenter (and I) are talking about the pages that malloc keeps around in the processes memory to be reused (And thus are never released back to the OS). Are you saying FreeBSD/OpenBSD/others have a system to tell the kernel when a user process has pages it plans to zero, and then a system for the kernel to notify the process when/if it does? That would be pretty interesting to see, but I've never heard of that being a thing.
      - masklinn 11y ago
        > Would you mind providing some more information on this? I googled it but I couldn't find any documentation on what you describing. I was talking about https://www.freebsd.org/doc/en/articles/vm-design/prefault-optimizations.html https://www.freebsd.org/doc/en/articles/vm-design/prefault-o... but after actually re-checking the docs rather than going from bitrotted memory pre-zeroing only happens for the initial zero-fill fault, so the original calloc is "free" (if there are zeroed pages in the queue) but freeing then re-calloc'ing will need to zero the chunk (unless the allocator decides to go see if the kernel has pre-zeroed space instead I guess). My bad. > To be clear though, the parent commenter (and I) are talking about the pages that malloc keeps around in the processes memory to be reused (And thus are never released back to the OS). Yeah so was I, but I was misremembering stuff. > Are you saying FreeBSD/OpenBSD/others have a system to tell the kernel when a user process has pages it plans to zero, and then a system for the kernel to notify the process when/if it does? Turns out no.