3 ms·

Anyone who's written a modicum of C would agree with that. I've written tens of thousands of lines of C and maintain a codebase of over 50KLOC. I also am a so

Anyone who's written a modicum of C would agree with that. I've written tens of thousands of lines of C and maintain a codebase of over 50KLOC. I also am a somewhat less active maintainer on another project with almost 400KLOC. And I would strongly disagree with that, so you shouldn't make wide generalizations like "anyone." I have devoted probably less than an hour of my time out of over a year of development on this codebase to memory management. I have never understood why people think it is so hard to call free() a couple times when your program shuts down. At least in my experience, if you're finding memory management to be "hard", your program is structured in a bad way to begin with. Your program shouldn't be covered in mallocs and frees.

statictype 18y ago
>I have devoted probably less than an hour of my time out of over a year of development on this codebase to memory management. I have never understood why people think it is so hard to call free() a couple times whenhttp://news.ycombinator.com/item?id=435267 http://news.ycombinator.com/item?id=435267 your program shuts down. I'm completely and utterly unable to comprehend how a code-base of the size you mentioned could have its memory managed by a couple of free() calls when your program shuts down. (And if you're program is shutting down, why would you even need to call free() at all? ) Are you using some kind of garbage collector tacked on top of C? Edit: Can you elaborate on the nature of these programs? Because if you're building any sort of non-trivial data structures in your code at all, then you almost certainly need allocations and deallocations
- DarkShikari 18y ago
  We have to call free when the program shuts down because the program is available through a library interface, and so it can be called as a linked library in addition to being used as a standalone program. If it wasn't for that, we wouldn't even need free. The program is structured so that based on the parameters it is given, we know exactly what needs to be allocated: how many video frames need to be malloced and so forth. This is all done in the initialization functions and after the program has started, it never needs to grab any memory ever again. This is not a case of "we're allocating the max we need and distributing it later"--it's a case of since we know how the program works, we know what exactly what it will need to do and how many frames it will do it to simultaneously. There are only four things in the entire program that are malloced: 1. Video frame objects (big structs with tons of pointers to data that goes along with a video frame). A new_frame function exists to malloc and return a video frame. This makes it easy to check mallocs. We know exactly how many frames will be needed on startup. These frames make up the vast majority of memory usage of the program. 2. One data struct for each thread. Each thread has its own local struct that it passes around containing data currently being worked on. 3. A scratch buffer, per thread. This is a small malloced buffer used for a few calculations that require variable-size buffers, usually depending on video frame width (i.e. something known at initialization, but not known at compile-time). 4. Bitstream output buffers for each thread. In the extremely rare case that a bitstream exceeds the initial allocated value, this is realloced on demand. Every single one of these can be free'd trivially at the end of runtime.
  - lacker 18y ago
    Well, of course you find malloc easy if you know your exact memory requirements when your code starts. If you are writing something like a long-running server process, you will be more able to appreciate garbage collection.
  - statictype 18y ago
    Interesting. Thanks for the details. I'm guessing this is a non-interactive library\application which takes input at start-up, does work and then exits? (because otherwise, I assume it would be impossible to reason about how much memory is required)
    - DarkShikari 18y ago
      The program is x264 and it can be interactive to some extent--it has a "reconfig" function that allows you to reconfigure the parameters on the fly, though it has some limitations as to what you can adjust and what you cannot. One potential purpose of this is a "speed control" feature where you adjust settings on the fly to ensure realtime operation, such as for a broadcast encoding server (my company does this).
- lbrandy 18y ago
  If you are writing good, and highly reliable C, that is meant to run for long period of time, there will be no mallocs() and frees() inside the main loop because memory fragmentation can eventually crash your program. You will dynamically allocate and re-use memory outside of the main loop. You will use the stack for temporary creations. You can actually calculate the maximal stack usage, and ensure you have enough space, instead of 10 days into running that malloc() decides to return a null pointer because of fragmentation. And one (very good) reason you call free() before your program terminates is so that you can use tools to figure out if there are any "real" memory leaks in your code. This way you don't have to wade through piles and piles of "oh that's not a 'real' memory leak because it'll get cleaned up when the program terminates".

statictype 18y ago

>I have devoted probably less than an hour of my time out of over a year of development on this codebase to memory management. I have never understood why people think it is so hard to call free() a couple times whenhttp://news.ycombinator.com/item?id=435267 http://news.ycombinator.com/item?id=435267 your program shuts down. I'm completely and utterly unable to comprehend how a code-base of the size you mentioned could have its memory managed by a couple of free() calls when your program shuts down. (And if you're program is shutting down, why would you even need to call free() at all? ) Are you using some kind of garbage collector tacked on top of C? Edit: Can you elaborate on the nature of these programs? Because if you're building any sort of non-trivial data structures in your code at all, then you almost certainly need allocations and deallocations

DarkShikari 18y ago
We have to call free when the program shuts down because the program is available through a library interface, and so it can be called as a linked library in addition to being used as a standalone program. If it wasn't for that, we wouldn't even need free. The program is structured so that based on the parameters it is given, we know exactly what needs to be allocated: how many video frames need to be malloced and so forth. This is all done in the initialization functions and after the program has started, it never needs to grab any memory ever again. This is not a case of "we're allocating the max we need and distributing it later"--it's a case of since we know how the program works, we know what exactly what it will need to do and how many frames it will do it to simultaneously. There are only four things in the entire program that are malloced: 1. Video frame objects (big structs with tons of pointers to data that goes along with a video frame). A new_frame function exists to malloc and return a video frame. This makes it easy to check mallocs. We know exactly how many frames will be needed on startup. These frames make up the vast majority of memory usage of the program. 2. One data struct for each thread. Each thread has its own local struct that it passes around containing data currently being worked on. 3. A scratch buffer, per thread. This is a small malloced buffer used for a few calculations that require variable-size buffers, usually depending on video frame width (i.e. something known at initialization, but not known at compile-time). 4. Bitstream output buffers for each thread. In the extremely rare case that a bitstream exceeds the initial allocated value, this is realloced on demand. Every single one of these can be free'd trivially at the end of runtime.
- lacker 18y ago
  Well, of course you find malloc easy if you know your exact memory requirements when your code starts. If you are writing something like a long-running server process, you will be more able to appreciate garbage collection.
- statictype 18y ago
  Interesting. Thanks for the details. I'm guessing this is a non-interactive library\application which takes input at start-up, does work and then exits? (because otherwise, I assume it would be impossible to reason about how much memory is required)
  - DarkShikari 18y ago
    The program is x264 and it can be interactive to some extent--it has a "reconfig" function that allows you to reconfigure the parameters on the fly, though it has some limitations as to what you can adjust and what you cannot. One potential purpose of this is a "speed control" feature where you adjust settings on the fly to ensure realtime operation, such as for a broadcast encoding server (my company does this).
lbrandy 18y ago
If you are writing good, and highly reliable C, that is meant to run for long period of time, there will be no mallocs() and frees() inside the main loop because memory fragmentation can eventually crash your program. You will dynamically allocate and re-use memory outside of the main loop. You will use the stack for temporary creations. You can actually calculate the maximal stack usage, and ensure you have enough space, instead of 10 days into running that malloc() decides to return a null pointer because of fragmentation. And one (very good) reason you call free() before your program terminates is so that you can use tools to figure out if there are any "real" memory leaks in your code. This way you don't have to wade through piles and piles of "oh that's not a 'real' memory leak because it'll get cleaned up when the program terminates".