4 ms·

Show HN: Flexible C memory allocation scheme with leak checking

52 points by mulle_nat 10y ago

JoachimSchipper 10y ago
This seems to be a malloc()/free() wrapper that keeps track of malloc()ed/realloc()ed/free()d pointers. I see that it's been submitted by the author, so one hopefully-helpful comment: it would be really helpful to highlight why one would use this instead of one of the well-established alternatives (Boehm GC, valgrind, ASan). The "run after every test case" example actually gives a fairly good reason, but note that realistic code bases probably have a few caches, freelist-optimizations or other exceptions to "deallocate all memory after every test".
- ktRolster 10y ago
  realistic code bases probably have a few caches, freelist-optimizations or other exceptions to "deallocate all memory after every test". If an open source library has memory allocated at finish, to me that's a warning to find a different project. If it's something I get assigned to work on at work, I still have to work on it; but it lets me know I'll probably be cleaning up many other things as well: the programmers who passed before were sloppy.
  - ars 10y ago
    > If an open source library has memory allocated at finish, to me that's a warning to find a different project. You need to read this: https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283 Basically it's faster to simply exit and not try to walk datastructures deallocating everything.
    - ktRolster 10y ago
      That rarely becomes an issue, and should not be optimized for by default.
      - eutectic 10y ago
        What's the advantage to deallocating on exit?
        ktRolster 10y ago
        You know your code is clean.
        luckydude 10y ago
        Hi, BitKeeper guy here (BK predates git, first distributed SCM). I built what I'm sure is a less good version of this, I called it purify, you included purify.h and it warned you about all the stuff that it could by redefining malloc/free/etc to ones that kept track of stuff. So some overlap with the system posted here. We were very careful to free everything before exit. It just felt cleaner. It was also a bad idea. Until a few years ago, BK used an ascii file format, which means it read in the graph with all the meta data, parsing that, malloc-ing each string, each struct, etc. At exit we carefully freed all that. Why was that a bad idea? If you wanted to link BK into your long lived IDE or wanted it to be a library, then it was a very good idea. But we never did that and BK doesn't really want to be used that way. So it became a performance problem. Consider bk changes -1 # same as git log -1 That used to parse the entire changeset graph, build an in memory one version of it, find the tip (HEAD), print out the comment, then recursively free everything. All for the holy grail of "knowing your code is clean". Clean is good unless it is pointless. Another way to say the same thing is you could make the claim that all code should be written to be thread safe. Even if you never multi thread it. That's a lot of work, maybe not so bad for people who are good at threading, but it's work for most people (I'm a former Sun kernel hack, I get threads, it's not that hard once you get over the hump but a lot of people don't). If you have no intention of multi threading the code then making it thread safe is just "clean code" masturbation. I'm all for clean code stuff where you are going to reap some value. In our case we did all that clean code memory management and eventually said screw that, changed the file format so we could mmap() it and page it in (read/write protect everything, take the fault, fault handler pages it in). No alloc, no free, way better performance. All those years of clean code bought us nothing. I get that if you are making a library the rules about clean are very very different. But for an app that is going to run and exit, don't push clean down people's throat, it isn't helping in a lot of cases.
    - mulle_nat 10y ago
      I think one should differentiate between app/tool code and library/plugin/daemon code. A typical tool like git starts, runs for a while and then exits. In this case worrying too much about memory cleanness is counterproductive. This changes when you are writing a data structure, like say a hashtable. You don't want a hashtable that leaks during operation. It changes when you write a daemon, that runs for a long time, you don't want it to consume all address space. It changes when you write plugins, that get repeatedly loaded and unloaded.
- mulle_nat 10y ago
  An advantage over valgrind f.e. is that the same test works on Windows to Linux to OS X without having to interface with different tools. The use of the "allocator" struct enables you to isolate your code to test, so that other caches do not obscure the result. The thing is, it's not just a wrapper, but also a decoupler.
ktRolster 10y ago
1) The documentation doesn't show what file needs to be included; I think it's "mulle_allocator/mulle_allocator.h" 2) It can be convenient, in mulle_allocator.h, to have a define: #define malloc() mulle_malloc(). That way you don't actually need to dig through the code and replace anything. 3) If you're doing #defines, you can do something like this: #define malloc(a) mulle_malloc(a, __LINE__, __FILE__) which will let you keep track of the line and file where the memory was allocated. 4) If you're on Linux (which often C programmers are not) then mtrace() works really nicely for this. 5) It's always better to have more options for memory debugging, so good job!
- mulle_nat 10y ago
  I will update the documentation with the suggestion. I would not want to place the malloc #define in mulle_allocator.h though. First, it's too surprising. And second, it defeats the possibilty to isolate certain memory operations.
IpV8 10y ago
We use something similar to this with object creation/deletion too. We set up a macro called watchallocation and watchdelete that keeps track of when objects get new'ed and deleted so we can track memory leaks. In release mode they just vanish. Quite helpfull
zde 10y ago
How do you redirect strdup() et al to your malloc? What's wrong with malloc hooks? http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html http://www.gnu.org/software/libc/manual/html_node/Hooks-for-...
- mulle_nat 10y ago
  Well for strdup, there is mulle_strdup. Who is Al ? :) There is nothing wrong with malloc hooks, but they are global and you can't isolate on a per data structure basis. Also when you malloc you know it's malloc, but calling mulle_allocator_malloc it could be something else (your own memory manager). That's the decoupling aspect.

52 points by mulle_nat 10y ago

This seems to be a malloc()/free() wrapper that keeps track of malloc()ed/realloc()ed/free()d pointers. I see that it's been submitted by the author, so one hopefully-helpful comment: it would be really helpful to highlight why one would use this instead of one of the well-established alternatives (Boehm GC, valgrind, ASan). The "run after every test case" example actually gives a fairly good reason, but note that realistic code bases probably have a few caches, freelist-optimizations or other exceptions to "deallocate all memory after every test".

ktRolster 10y ago
realistic code bases probably have a few caches, freelist-optimizations or other exceptions to "deallocate all memory after every test". If an open source library has memory allocated at finish, to me that's a warning to find a different project. If it's something I get assigned to work on at work, I still have to work on it; but it lets me know I'll probably be cleaning up many other things as well: the programmers who passed before were sloppy.
- ars 10y ago
  > If an open source library has memory allocated at finish, to me that's a warning to find a different project. You need to read this: https://news.ycombinator.com/item?id=8305283 https://news.ycombinator.com/item?id=8305283 Basically it's faster to simply exit and not try to walk datastructures deallocating everything.
  - ktRolster 10y ago
    That rarely becomes an issue, and should not be optimized for by default.
    - eutectic 10y ago
      What's the advantage to deallocating on exit?
      - ktRolster 10y ago
        You know your code is clean.
        luckydude 10y ago
        Hi, BitKeeper guy here (BK predates git, first distributed SCM). I built what I'm sure is a less good version of this, I called it purify, you included purify.h and it warned you about all the stuff that it could by redefining malloc/free/etc to ones that kept track of stuff. So some overlap with the system posted here. We were very careful to free everything before exit. It just felt cleaner. It was also a bad idea. Until a few years ago, BK used an ascii file format, which means it read in the graph with all the meta data, parsing that, malloc-ing each string, each struct, etc. At exit we carefully freed all that. Why was that a bad idea? If you wanted to link BK into your long lived IDE or wanted it to be a library, then it was a very good idea. But we never did that and BK doesn't really want to be used that way. So it became a performance problem. Consider bk changes -1 # same as git log -1 That used to parse the entire changeset graph, build an in memory one version of it, find the tip (HEAD), print out the comment, then recursively free everything. All for the holy grail of "knowing your code is clean". Clean is good unless it is pointless. Another way to say the same thing is you could make the claim that all code should be written to be thread safe. Even if you never multi thread it. That's a lot of work, maybe not so bad for people who are good at threading, but it's work for most people (I'm a former Sun kernel hack, I get threads, it's not that hard once you get over the hump but a lot of people don't). If you have no intention of multi threading the code then making it thread safe is just "clean code" masturbation. I'm all for clean code stuff where you are going to reap some value. In our case we did all that clean code memory management and eventually said screw that, changed the file format so we could mmap() it and page it in (read/write protect everything, take the fault, fault handler pages it in). No alloc, no free, way better performance. All those years of clean code bought us nothing. I get that if you are making a library the rules about clean are very very different. But for an app that is going to run and exit, don't push clean down people's throat, it isn't helping in a lot of cases.
  - mulle_nat 10y ago
    I think one should differentiate between app/tool code and library/plugin/daemon code. A typical tool like git starts, runs for a while and then exits. In this case worrying too much about memory cleanness is counterproductive. This changes when you are writing a data structure, like say a hashtable. You don't want a hashtable that leaks during operation. It changes when you write a daemon, that runs for a long time, you don't want it to consume all address space. It changes when you write plugins, that get repeatedly loaded and unloaded.
mulle_nat 10y ago
An advantage over valgrind f.e. is that the same test works on Windows to Linux to OS X without having to interface with different tools. The use of the "allocator" struct enables you to isolate your code to test, so that other caches do not obscure the result. The thing is, it's not just a wrapper, but also a decoupler.

ktRolster 10y ago

1) The documentation doesn't show what file needs to be included; I think it's "mulle_allocator/mulle_allocator.h" 2) It can be convenient, in mulle_allocator.h, to have a define: #define malloc() mulle_malloc(). That way you don't actually need to dig through the code and replace anything. 3) If you're doing #defines, you can do something like this: #define malloc(a) mulle_malloc(a, __LINE__, __FILE__) which will let you keep track of the line and file where the memory was allocated. 4) If you're on Linux (which often C programmers are not) then mtrace() works really nicely for this. 5) It's always better to have more options for memory debugging, so good job!

mulle_nat 10y ago
I will update the documentation with the suggestion. I would not want to place the malloc #define in mulle_allocator.h though. First, it's too surprising. And second, it defeats the possibilty to isolate certain memory operations.

IpV8 10y ago

We use something similar to this with object creation/deletion too. We set up a macro called watchallocation and watchdelete that keeps track of when objects get new'ed and deleted so we can track memory leaks. In release mode they just vanish. Quite helpfull

zde 10y ago

How do you redirect strdup() et al to your malloc? What's wrong with malloc hooks? http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html http://www.gnu.org/software/libc/manual/html_node/Hooks-for-...

mulle_nat 10y ago
Well for strdup, there is mulle_strdup. Who is Al ? :) There is nothing wrong with malloc hooks, but they are global and you can't isolate on a per data structure basis. Also when you malloc you know it's malloc, but calling mulle_allocator_malloc it could be something else (your own memory manager). That's the decoupling aspect.