14 ms·

Are there functions available with which I can at run-time query how much OS memory is used, how much handed out in allocations, how many mmap()ed pools are use

by nh2 7y ago

Are there functions available with which I can at run-time query how much OS memory is used, how much handed out in allocations, how many mmap()ed pools are used, and so on? I find that one of the most important features of a malloc library to debug memory usage. glibc has these functions (like malloc_info()) -- they are very bugged in that they return wrong results, but after patching them to be correct, they are super useful.

jasonzemos 7y ago
Any more info/link on how they are incorrect and the patches you need to fix them?
- nh2 7y ago
  Sure, my patches are linked in the bugs I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=24026 https://sourceware.org/bugzilla/show_bug.cgi?id=24026 - "malloc_info() returns wrong numbers" https://sourceware.org/bugzilla/show_bug.cgi?id=21556 https://sourceware.org/bugzilla/show_bug.cgi?id=21556 - "malloc_stats printing size_t fields as unsigned int" If you're interested in this topic: After finding one bug after the other due to 32-bit integer overflow in malloc.c, I just searched for "unsigned int" in that file for fun, and 30 seconds later found what I consider a security vulnerability in realloc(): https://sourceware.org/bugzilla/show_bug.cgi?id=24027 https://sourceware.org/bugzilla/show_bug.cgi?id=24027 In certain situations, if you realloc() e.g. 32G + 5 bytes (reallocs this large can happen in large programs e.g. for data analysis), it'll copy only 5 bytes, and leave the rest as memory garbage. That experience taught me that open source code being old doesn't mean anybody ever read it.
  - rurban 7y ago
    pt2malloc is only maintained by glibc, but not really. Since they cannot maintain it, they want to get rid of it. The upstream maintainer released a better version pt3malloc, which glibc refused to adopt. It needs one more word per alloc.
ndesaulniers 7y ago
On Linux, there's a bit of the info you're looking for under /proc/self/*. (Not a great answer; the info under /proc/ isn't exactly well structured, but doesn't change super often).
- nh2 7y ago
  /proc doesn't really tell you much about userspace concerns such as "how much memory has the malloc implementation handed out to the program. It describes only the kernel-userspace parts of memory usage. For example, malloc fragmentation is not inspectable in it. (If you enable the mallopt() setting that turns each malloc() into a separate mmap(), then it tells you a lot, but that is not a default setting.)

by nh2 7y ago

jasonzemos 7y ago

Any more info/link on how they are incorrect and the patches you need to fix them?

nh2 7y ago
Sure, my patches are linked in the bugs I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=24026 https://sourceware.org/bugzilla/show_bug.cgi?id=24026 - "malloc_info() returns wrong numbers" https://sourceware.org/bugzilla/show_bug.cgi?id=21556 https://sourceware.org/bugzilla/show_bug.cgi?id=21556 - "malloc_stats printing size_t fields as unsigned int" If you're interested in this topic: After finding one bug after the other due to 32-bit integer overflow in malloc.c, I just searched for "unsigned int" in that file for fun, and 30 seconds later found what I consider a security vulnerability in realloc(): https://sourceware.org/bugzilla/show_bug.cgi?id=24027 https://sourceware.org/bugzilla/show_bug.cgi?id=24027 In certain situations, if you realloc() e.g. 32G + 5 bytes (reallocs this large can happen in large programs e.g. for data analysis), it'll copy only 5 bytes, and leave the rest as memory garbage. That experience taught me that open source code being old doesn't mean anybody ever read it.
- rurban 7y ago
  pt2malloc is only maintained by glibc, but not really. Since they cannot maintain it, they want to get rid of it. The upstream maintainer released a better version pt3malloc, which glibc refused to adopt. It needs one more word per alloc.

ndesaulniers 7y ago

On Linux, there's a bit of the info you're looking for under /proc/self/*. (Not a great answer; the info under /proc/ isn't exactly well structured, but doesn't change super often).

nh2 7y ago
/proc doesn't really tell you much about userspace concerns such as "how much memory has the malloc implementation handed out to the program. It describes only the kernel-userspace parts of memory usage. For example, malloc fragmentation is not inspectable in it. (If you enable the mallopt() setting that turns each malloc() into a separate mmap(), then it tells you a lot, but that is not a default setting.)