5 ms·
In my first job I work on a database product in development that leaked memory slowly, leading to crashes after hours of usage. The software was written in C an
by ww520 4y ago
In my first job I work on a database product in development that leaked memory slowly, leading to crashes after hours of usage. The software was written in C and there were no tools like Purify or Valgrind back then to deal with memory problem. It was a vexing problem that got punted until release time, when it became a show stopper.
I looked into the problem and found that the memory allocation used malloc and free. I then defined macros for malloc and free in a common header to call my_malloc and my_free functions with the standard __FILE__ and __LINE__ macros passed in as parameters. Re-compiled the whole program with the macros, which redirected every call of malloc and free to my functions. My functions logged all the calls with the allocated memory pointer, the filename, and the line number. Once I collected enough log data from test runs, I sorted the lines in the log file by the memory pointer address. Every pointer address should come in pair, one from the malloc() and one from the free(). The odd number pointer addresses are the ones with missing free(). And I got their filename and line number right there.
- bane 4y agoYou know, that's surprisingly an elegant hack. I'm sure it's obvious to some folks, but for some reason I never came up with it during my old C programming days.
- ww520 4y agoOh I was desperate. I was the most junior guy in the team on my first job after school whom got thrown with a hard problem. Luckily I remembered __FILE__ and __LINE__ in C, and it was a matter of working backward to link those to malloc and free somehow. My Unix command-fu (sort, uniq -c) learned back in school came through in dealing with the huge log file. It was a huge boost to my confidence on my job and earned my cred with the team.
- bee_rider 4y agoThe eternal curse of the C programmer is knowing that macros are Right There, just waiting for you to take them up and craft the most elegant… footgun.
- josephg 4y agoHah! A few years ago I did something similar. I was in the process of porting some C code (a physics engine) to javascript. After porting the code I benchmarked it - and, no surprises - it was waay slower than the original C code. One reason C is faster than javascript is that C structs get "inlined" into the containing object. For example, in C struct Body { vec2 pos; vec2 velocity; } would just be 1 object. But the equivalent javascript code would allocate 3 objects instead. I inlined vec2 (replacing it with pos_x, pos_y, etc) and performance got a lot better. But I was curious what other structs were thrashing the garbage collector. So I added a call into all my constructors which generated a stack trace (new Error().stack), and used the stack trace as the key in a javascript object - with the value being the number of times that stack trace was seen. After sorting and printing the result, I had a hit list of the hottest stack traces which were thrashing V8's garbage collector. I fixed all the worst call sites and by the time I was done performance improved by about 3x or so!
- gchaincl 4y agoThis is awesome, i had to do the same, ended up writing a .h file that redefined malloc/free with macros, and then reported the missing and double frees. https://github.com/qustavo/smc/blob/master/smc.h#L191 https://github.com/qustavo/smc/blob/master/smc.h#L191
- stingraycharles 4y agoOh man this brings back memories, and is the exact same approach I used a couple of decades ago — there was some undefined behaviour in some code, and it ended up being nearly impossible to reproduce when using valgrind. Using the approach you described is extremely effective and low-overhead, and allowed me to actually reproduce the issue (I think in the end it was due some async / race condition issue, because I was using “if map[key]” in a place where I shouldn’t, which actually inserts the key if it’s missing (and thus mutates the map). Fun times.
- saagarjha 4y agoThis is a great strategy even today, with the only difference being how you'd implement it: typically by interposing malloc and free and having it walk its own stack.
- jzer0cool 4y agoAny good tools or approach similarly for java applications? Those which crashes due to memory leaks?