5 ms·

But desktop processors also don't zero pages, do they? And by zeroing pages, you prevent lazy memory allocation policies which do not allocate the pages until

But desktop processors also don't zero pages, do they? And by zeroing pages, you prevent lazy memory allocation policies which do not allocate the pages until they're read or written for the first time. This can have a significant impact on memory usage, data locality etc. So I'm quite skeptical that there is no difference between calloc and malloc. Is there more evidences of this somewhere?

kqr 11y ago
How does it prevent such allocation? Can you not just mark the first read from an area to always return 0, and then allocate as soon as it happens? That also has the benefit of not having to zero anything if you're only writing to it.
- DSMan195276 11y ago
  Internally, an OS like Linux maps every page in a requested piece of memory to a read-only page of zeros. When you attempt to read from this page, it works fine. When you attempt to write to it, the MMU causes a page-fault and the OS inserts a page where the zero-page is, and then continues your program execution. Thus, it doesn't actually have to allocate any memory for you if you never write to the new pages. But the OS/MMU doesn't distinguish between a regular write, and a write of zero. Thus, if you manually zero every page you get (And thus write zeros to the read-only zero-page), it'll page-fault back to the OS and the OS will have to allocate a new page so that the write succeeds - Even though if you didn't do the zeroing of memory you would have gotten the same effect of having a bunch of zeros in memory, but without having to allocate any new pages for your process.
  - kqr 11y ago
    Isn't that just saying that calloc is compatible with lazy allocation?
    - DSMan195276 11y ago
      Kinda. Reading my comment a second time, I'm not exactly happy with my description, since while it's 'right' is a very simplistic description, ignoring some of the finer points. Since malloc/calloc are generally used for smaller allocations, the chances you can actually avoid allocating some pages you ask for is pretty slim since a bunch of objects get placed into the same page (And thus writing to any of them will trigger a new page being allocated). There's also no guarantee there isn't headers for malloc to make use of, or similar surrounding your piece of memory, which makes the point moot - Just using malloc triggers writes to at least the first page. So while calloc/malloc are kinda compatible with lazy-allocation, you really shouldn't rely on it being a thing, and it probably won't matter. It's worth understanding, but the chances it actually comes into play aren't huge. If your program does lots of small malloc's and free's, then it basically won't matter because you won't be asking the kernel for more memory, just reusing what you already have. If you care about taking advantage of lazy-allocation for one reason or another, the bottom line is probably that you shouldn't be using malloc and calloc for that then. Just use mmap directly and you'll have a better time - more control, you have a nice page-aligned address to start with, and you can be sure the memory is untouched. malloc and calloc are good for general allocations, but using mmap takes out the guesswork when you have something big and specific you need to allocate.
umanwizard 11y ago
Desktop OSs (not processors afaik) zero pages. To not do so would leak information between privilege contexts. E.g. imagine /etc/passwd is read and later on the page it occupied is put back in the free pool. Another process comes along and asks for memory. It gets that page and can now read /etc/passwd .
- x0 11y ago
  Bit lost here, not really a C programmer, by "zeroing pages" are we talking about zeroing out memory? Because I wrote a little C program that just reads memory until it segfaults, usually I can get about 3gb of stuff out, random little bits of files. I've noticed it works much better on OS X than Linux.
  - umanwizard 11y ago
    ...what? Uh, I don't know how memory allocation works in detail in OSX, so you could be right, but it'd surprise me. Can you show me your program?
    - x0 11y ago
      here it is: https://gist.github.com/raincoats/7b2631106c74f759eb5d https://gist.github.com/raincoats/7b2631106c74f759eb5d A few things though: - Last time I ran this, in the memory dump, every second character was the letter "f". Like, in the dump, instead of saying "Words" it would say "Wfofrfdfsf". It changes each time, probably to do with my char onechar variable. (Actually now that I think about it, it should probably only be onechar[1] instead of onechar[2]. It was called onechar because I was trying to printf 1 char at a time) - Before I changed "p" to &memcpy, it was "p = &p", and it would only read my environ before segfaulting. - This program, on linux, dies almost immediately. It only spits out a few bytes.
      - cowsandmilk 11y ago
        your program doesn't allocate memory, so it is unrelated to memory allocation. You instead are taking the address of a function in memory (memcpy); and then printing out the function. This isn't a leak in any way since your program has to be able to reach memcpy and call it. So, no, you aren't printing out bits of files. You're printing out functions that your program has the ability to call.

kqr 11y ago

How does it prevent such allocation? Can you not just mark the first read from an area to always return 0, and then allocate as soon as it happens? That also has the benefit of not having to zero anything if you're only writing to it.

DSMan195276 11y ago
Internally, an OS like Linux maps every page in a requested piece of memory to a read-only page of zeros. When you attempt to read from this page, it works fine. When you attempt to write to it, the MMU causes a page-fault and the OS inserts a page where the zero-page is, and then continues your program execution. Thus, it doesn't actually have to allocate any memory for you if you never write to the new pages. But the OS/MMU doesn't distinguish between a regular write, and a write of zero. Thus, if you manually zero every page you get (And thus write zeros to the read-only zero-page), it'll page-fault back to the OS and the OS will have to allocate a new page so that the write succeeds - Even though if you didn't do the zeroing of memory you would have gotten the same effect of having a bunch of zeros in memory, but without having to allocate any new pages for your process.
- kqr 11y ago
  Isn't that just saying that calloc is compatible with lazy allocation?
  - DSMan195276 11y ago
    Kinda. Reading my comment a second time, I'm not exactly happy with my description, since while it's 'right' is a very simplistic description, ignoring some of the finer points. Since malloc/calloc are generally used for smaller allocations, the chances you can actually avoid allocating some pages you ask for is pretty slim since a bunch of objects get placed into the same page (And thus writing to any of them will trigger a new page being allocated). There's also no guarantee there isn't headers for malloc to make use of, or similar surrounding your piece of memory, which makes the point moot - Just using malloc triggers writes to at least the first page. So while calloc/malloc are kinda compatible with lazy-allocation, you really shouldn't rely on it being a thing, and it probably won't matter. It's worth understanding, but the chances it actually comes into play aren't huge. If your program does lots of small malloc's and free's, then it basically won't matter because you won't be asking the kernel for more memory, just reusing what you already have. If you care about taking advantage of lazy-allocation for one reason or another, the bottom line is probably that you shouldn't be using malloc and calloc for that then. Just use mmap directly and you'll have a better time - more control, you have a nice page-aligned address to start with, and you can be sure the memory is untouched. malloc and calloc are good for general allocations, but using mmap takes out the guesswork when you have something big and specific you need to allocate.

umanwizard 11y ago

Desktop OSs (not processors afaik) zero pages. To not do so would leak information between privilege contexts. E.g. imagine /etc/passwd is read and later on the page it occupied is put back in the free pool. Another process comes along and asks for memory. It gets that page and can now read /etc/passwd .

x0 11y ago
Bit lost here, not really a C programmer, by "zeroing pages" are we talking about zeroing out memory? Because I wrote a little C program that just reads memory until it segfaults, usually I can get about 3gb of stuff out, random little bits of files. I've noticed it works much better on OS X than Linux.
- umanwizard 11y ago
  ...what? Uh, I don't know how memory allocation works in detail in OSX, so you could be right, but it'd surprise me. Can you show me your program?
  - x0 11y ago
    here it is: https://gist.github.com/raincoats/7b2631106c74f759eb5d https://gist.github.com/raincoats/7b2631106c74f759eb5d A few things though: - Last time I ran this, in the memory dump, every second character was the letter "f". Like, in the dump, instead of saying "Words" it would say "Wfofrfdfsf". It changes each time, probably to do with my char onechar variable. (Actually now that I think about it, it should probably only be onechar[1] instead of onechar[2]. It was called onechar because I was trying to printf 1 char at a time) - Before I changed "p" to &memcpy, it was "p = &p", and it would only read my environ before segfaulting. - This program, on linux, dies almost immediately. It only spits out a few bytes.
    - cowsandmilk 11y ago
      your program doesn't allocate memory, so it is unrelated to memory allocation. You instead are taking the address of a function in memory (memcpy); and then printing out the function. This isn't a leak in any way since your program has to be able to reach memcpy and call it. So, no, you aren't printing out bits of files. You're printing out functions that your program has the ability to call.