4 ms·

Yes, if it was the custom malloc that allowed memory issues to go undetected, what else is doing that?

SwellJoe 12y ago
The Squid codebase had a custom malloc in it for a while for some operating systems, but on systems with competent malloc it used the OS implementation. I doubt that malloc remains today (I believe Squid has been somewhat rewritten in C++, with a more modern approach to memory allocation, since I was working on it). There was a time about 15 years ago, when many UNIX variants had sub-optimal malloc, or other core libraries, and custom implementations were common in large Open Source projects (as well as commercial products). Those operating systems are mostly gone (Solaris remains, kinda, and there's still some big iron running those old beasts), but I haven't seen a newish project using a custom malloc or other core library in years. I couldn't name the projects that made that choice back then, but I remember seeing it very frequently, when I used to read a lot more C source than I do, today.
- Someone 12y ago
  Webkit has a custom allocator. See http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/bmalloc.h http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/..., and, via, http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/Cache.cpp http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/..., http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/VMAllocate.h http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/..., which seems to use mmap and a SYSCALL macro. That doesn't seem to build on the system's alloc. According to https://www.webkit.org/quality/leakhunting.html https://www.webkit.org/quality/leakhunting.html, it may even have multiple ones: "Get a fresh WebKit build, ideally a Debug build as this turns off the various custom allocators in WebKit." It wouldn't surprise me to find custom allocations in many other large projects that may be memory constrained (postgres, open office, gcc, clang, you name it) at some size, the extra burden of a custom allocator isn't that large relative to the size of a project.
  - chrisrohlf 12y ago
    Yes, exactly this. Many large programs have custom allocators for performance and most of them are simple primitives built on top of chunked up pages returned by mmap/VirtualAlloc. The examples of this are all over. This obsession with shaming the OpenSSL developers for wrapping malloc is approaching unhealthy levels.
chrisrohlf 12y ago
Im not sure why this misunderstanding of the code continues to persist. It is not a custom malloc. It is a custom API for reusing chunks of a specific size previously returned by malloc.

The Squid codebase had a custom malloc in it for a while for some operating systems, but on systems with competent malloc it used the OS implementation. I doubt that malloc remains today (I believe Squid has been somewhat rewritten in C++, with a more modern approach to memory allocation, since I was working on it). There was a time about 15 years ago, when many UNIX variants had sub-optimal malloc, or other core libraries, and custom implementations were common in large Open Source projects (as well as commercial products). Those operating systems are mostly gone (Solaris remains, kinda, and there's still some big iron running those old beasts), but I haven't seen a newish project using a custom malloc or other core library in years. I couldn't name the projects that made that choice back then, but I remember seeing it very frequently, when I used to read a lot more C source than I do, today.

Someone 12y ago
Webkit has a custom allocator. See http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/bmalloc.h http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/..., and, via, http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/Cache.cpp http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/..., http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/VMAllocate.h http://trac.webkit.org/browser/trunk/Source/bmalloc/bmalloc/..., which seems to use mmap and a SYSCALL macro. That doesn't seem to build on the system's alloc. According to https://www.webkit.org/quality/leakhunting.html https://www.webkit.org/quality/leakhunting.html, it may even have multiple ones: "Get a fresh WebKit build, ideally a Debug build as this turns off the various custom allocators in WebKit." It wouldn't surprise me to find custom allocations in many other large projects that may be memory constrained (postgres, open office, gcc, clang, you name it) at some size, the extra burden of a custom allocator isn't that large relative to the size of a project.
- chrisrohlf 12y ago
  Yes, exactly this. Many large programs have custom allocators for performance and most of them are simple primitives built on top of chunked up pages returned by mmap/VirtualAlloc. The examples of this are all over. This obsession with shaming the OpenSSL developers for wrapping malloc is approaching unhealthy levels.

chrisrohlf 12y ago

Im not sure why this misunderstanding of the code continues to persist. It is not a custom malloc. It is a custom API for reusing chunks of a specific size previously returned by malloc.