3 ms·

> When an application frees an object it always has the original, requested allocation size on hand. After all, it’s a necessary condition to use the object cor

by kccqzy 3y ago

> When an application frees an object it always has the original, requested allocation size on hand. After all, it’s a necessary condition to use the object correctly. This is quite untrue in real-world C codebases unless you have a lot of discipline. I know from reading a previous article (https://nullprogram.com/blog/2023/10/08/ https://nullprogram.com/blog/2023/10/08/) by this author that they hate null-terminated strings. But in your typical C code, you often allocate a bunch of memory to be used as a null-terminated string, return it or pass it to another function transferring ownership, and expect a different function to free() it. The length of the string is not necessarily the allocation size! The code could have allocated some predetermined max size for the buffer, build up the string gradually, and then add a null terminator. Thus, the function doing the free() can't pass the right size.

kragen 3y ago
if i have a function foo which calls functions bar and baz to build up a dynamically allocated string (using a custom allocator), then calls function quux to transfer ownership of the string, foo needs to store the information about how big the allocation currently is, and how to allocate more of it, in a place where functions bar and baz can access it. if it doesn't do that, bar or baz can easily overflow the buffer because they don't know how big it is. this is a particular case of 'having the requested allocation size on hand is a necessary condition to use the object correctly' but if foo is passing this data structure to bar and baz, it can pass it to quux too, so the problem as you described it just doesn't exist (unless your code is only working by the merest of chances and will have some cves later this year the first time some competent programmers look at it) 'bar or baz can easily overflow the buffer because they don't know how big it is' is a particular case of 'having the requested allocation size on hand is a necessary condition to use the object correctly'
- comex 3y ago
  There are many reasons it might be inconvenient for foo to pass that data to quux. Perhaps it’s actually quux that called foo rather than the other way around (in other words, foo is passing ownership through its return value rather than through a function argument), in which case C makes it easier to return one value than two. Or perhaps there’s just a lot of existing logic that passes owned strings around without an accompanying size parameter, which is perfectly safe to do if you’re not planning to ever mutate them. Of course you could easily design your codebase to not have that issue, if you’re writing it from scratch. But that’s not the question.
  - kragen 3y ago
    i agree that parts of the code that are not changing the length of the string can use it safely without knowing the buffer size it's been easy to return a struct by value in c for over 30 years, though in very early versions of c, structs in rvalue context implicitly decayed to pointers, like arrays. it's been efficient for about 20 years so it's not easier to return one value than two if you've already defined the struct type, which is presumably the case here there's people who avoid it because they're imitating c code from the 01980s (or, charitably, performance-sensitive c code from the 01990s), or because they're using some horrible abomination like common lisp's cffi, but that's really quite marginal and even if you didn't originally design your codebase that way, you can certainly fix the design. it'll probably even make it easier to understand and maintain, because you're always passing around a stralloc or whatever, instead of sometimes passing around a stralloc and other times passing around a raw buffer pointer
    - lmm 3y ago
      If you're going to go to all the effort of rewriting your codebase anyway, why would you keep it in C?
      - kragen 3y ago
        you can change the calling interface of a single function without rewriting your entire codebase also there are a lot of things for which c is still the best option, though rust is becoming appealing for more of them all the time
    - varjag 3y ago
      Your personal coding preferences and discipline will switch the jobs together with you. The new hires on the project you left behind are going to have their own perspective (likely very wrong from your POV) on how things should be done. If it's an open source project, idiosyncratic code will only increase friction for contribution for little practical gain.
      - 3y ago
- kccqzy 3y ago
  You are describing an ideal scenario where code is written by experts and those experts did not take any shortcuts. Real world C code is unfortunately anything but that. I haven't had the luxury of only working on expertly-written C code which frankly also makes me hate C code. What if the function bar only appends a number to a string? Then the maximum length it needs could become embedded knowledge by the foo function ("10 characters is enough to store this uint32_t so no need to pass anything related to the allocation" thought the programmer who hated tedium). What if bar and baz are not allowed to grow the allocation because foo has already enough buffer to store the maximum possible size (say a path on Windows, 260 bytes)? What if the function quux is in a different third-party library whose API cannot be modified to take the allocation size?
  - kragen 3y ago
    well, i do agree that there's a lot of code out there that only works by the merest of chances and will have several cves opened the first time a competent programmer looks at it; the kinds of implicit thoughts and prayers you're describing (sweet jesus, pvg, that isn't a uint32_t you're formatting) are precisely what i was talking about but i'm not talking about code where the authors didn't take any shortcuts, just boringly competent c code with the regular quantity of bugs in it. i'm sorry you've been subjected to such torments as you report. but the author of this article isn't talking about how to write such astoundingly shitty code; he's talking about how to write good code if the function quux is in a different third-party library, its api probably cannot be modified to call the custom allocator either (my initial comment draft suggested that you might be talking about such a case, but then i realized that didn't make any sense and so i deleted it)
eqvinox 3y ago
Indeed, and more — there are code bases (ours, FRRouting) that do the opposite: expect the allocator to track sizes and invoke malloc_usable_size() to do our own (per-type-of-allocation grouped) accounting (to subtract on realloc/free; for adding it's not needed of course since you have the size already.)

by kccqzy 3y ago

kragen 3y ago

if i have a function foo which calls functions bar and baz to build up a dynamically allocated string (using a custom allocator), then calls function quux to transfer ownership of the string, foo needs to store the information about how big the allocation currently is, and how to allocate more of it, in a place where functions bar and baz can access it. if it doesn't do that, bar or baz can easily overflow the buffer because they don't know how big it is. this is a particular case of 'having the requested allocation size on hand is a necessary condition to use the object correctly' but if foo is passing this data structure to bar and baz, it can pass it to quux too, so the problem as you described it just doesn't exist (unless your code is only working by the merest of chances and will have some cves later this year the first time some competent programmers look at it) 'bar or baz can easily overflow the buffer because they don't know how big it is' is a particular case of 'having the requested allocation size on hand is a necessary condition to use the object correctly'

comex 3y ago
There are many reasons it might be inconvenient for foo to pass that data to quux. Perhaps it’s actually quux that called foo rather than the other way around (in other words, foo is passing ownership through its return value rather than through a function argument), in which case C makes it easier to return one value than two. Or perhaps there’s just a lot of existing logic that passes owned strings around without an accompanying size parameter, which is perfectly safe to do if you’re not planning to ever mutate them. Of course you could easily design your codebase to not have that issue, if you’re writing it from scratch. But that’s not the question.
- kragen 3y ago
  i agree that parts of the code that are not changing the length of the string can use it safely without knowing the buffer size it's been easy to return a struct by value in c for over 30 years, though in very early versions of c, structs in rvalue context implicitly decayed to pointers, like arrays. it's been efficient for about 20 years so it's not easier to return one value than two if you've already defined the struct type, which is presumably the case here there's people who avoid it because they're imitating c code from the 01980s (or, charitably, performance-sensitive c code from the 01990s), or because they're using some horrible abomination like common lisp's cffi, but that's really quite marginal and even if you didn't originally design your codebase that way, you can certainly fix the design. it'll probably even make it easier to understand and maintain, because you're always passing around a stralloc or whatever, instead of sometimes passing around a stralloc and other times passing around a raw buffer pointer
  - lmm 3y ago
    If you're going to go to all the effort of rewriting your codebase anyway, why would you keep it in C?
    - kragen 3y ago
      you can change the calling interface of a single function without rewriting your entire codebase also there are a lot of things for which c is still the best option, though rust is becoming appealing for more of them all the time
  - varjag 3y ago
    Your personal coding preferences and discipline will switch the jobs together with you. The new hires on the project you left behind are going to have their own perspective (likely very wrong from your POV) on how things should be done. If it's an open source project, idiosyncratic code will only increase friction for contribution for little practical gain.
    - 3y ago
kccqzy 3y ago
You are describing an ideal scenario where code is written by experts and those experts did not take any shortcuts. Real world C code is unfortunately anything but that. I haven't had the luxury of only working on expertly-written C code which frankly also makes me hate C code. What if the function bar only appends a number to a string? Then the maximum length it needs could become embedded knowledge by the foo function ("10 characters is enough to store this uint32_t so no need to pass anything related to the allocation" thought the programmer who hated tedium). What if bar and baz are not allowed to grow the allocation because foo has already enough buffer to store the maximum possible size (say a path on Windows, 260 bytes)? What if the function quux is in a different third-party library whose API cannot be modified to take the allocation size?
- kragen 3y ago
  well, i do agree that there's a lot of code out there that only works by the merest of chances and will have several cves opened the first time a competent programmer looks at it; the kinds of implicit thoughts and prayers you're describing (sweet jesus, pvg, that isn't a uint32_t you're formatting) are precisely what i was talking about but i'm not talking about code where the authors didn't take any shortcuts, just boringly competent c code with the regular quantity of bugs in it. i'm sorry you've been subjected to such torments as you report. but the author of this article isn't talking about how to write such astoundingly shitty code; he's talking about how to write good code if the function quux is in a different third-party library, its api probably cannot be modified to call the custom allocator either (my initial comment draft suggested that you might be talking about such a case, but then i realized that didn't make any sense and so i deleted it)

eqvinox 3y ago

Indeed, and more — there are code bases (ours, FRRouting) that do the opposite: expect the allocator to track sizes and invoke malloc_usable_size() to do our own (per-type-of-allocation grouped) accounting (to subtract on realloc/free; for adding it's not needed of course since you have the size already.)