7 ms·

> the program continued using the same pointer, not the realloc call result, since the old pointer did not change > In this context, it is a bug in the applica

> the program continued using the same pointer, not the realloc call result, since the old pointer did not change > In this context, it is a bug in the application This is a non-intuitive result by the normal lens C pointers are viewed by; these two pointers compared equal, how could using one differ from using the other? Pointer provenance rears its ugly head here though; one of those "pointers that have the same value"'s provenance is different to the other. By the standard, they _are_ allowed to be treated differently, and honestly the standard requires it. https://www.ralfj.de/blog/2020/12/14/provenance.html https://www.ralfj.de/blog/2020/12/14/provenance.html is pretty accessible, and shockingly on point: "just because two pointers point to the same address, does not mean they are equal in the sense that they can be used interchangeably". EDIT: spelling

mahkoh 2y ago
Provenance might be used as justification now but the actual rules are simpler and stricter. After freeing (or reallocing) a pointer, the application must not inspect the pointer value anymore. Even `new_pointer == old_pointer` is not allowed. IIRC, one justification for this was to account for systems with non-flat memory where inspecting the value of the old pointer might cause a processor exception.
- Scaevolus 2y ago
  It also makes sense for an allocator that can satisfy new allocations with recently freed allocations.
- RustyRussell 2y ago
  > Even `new_pointer == old_pointer` is not allowed. This is legal. But dereferencing old_pointer even after this check had passed is undefined.
  - anttihaapala 2y ago
    No, it is not, and it has never been OK for as long as C has been standardized by ANSI or ISO. C89 says that "The value of a pointer that refers to freed space is indeterminate." and that behavior is undefined "upon use ... of indeterminately-valued objects", hence a compiled program can e.g. behave as if `new_pointer == old_pointer` even though the object was relocated in memory.
    - JonChesterfield 2y ago
      That standard says free takes a void*, i.e. it takes the pointer by value, and that passing values by value to a function prevents the callee from changing them. More literally, your pointer value is copied into a register before the call into libc, so free can't change the value even if it wants to. Realloc can't either for the same reason. That provenance has come up with the premise that a ! = a for a 64bit integer value is an error in the formalism, specifically the language is deliberately inventing things that cannot be so on hardware.
      - layer8 2y ago
        Note that (a) pointer representations can contain bits representing the provenance that don’t participate in equality comparison, and (b) a “C implementation” in terms of the C standard includes the compiler, which means the compiler is allowed to track pointer provenance statically even when the representation in memory doesn’t differ (e.g. exactly what GCC is doing here). In addition, a C implementation is also allowed to track pointer provenance externally at runtime, again even if the pointer representation (as exposed by memcpy into a char buffer) is the same.
        JonChesterfield 2y ago
        To the extent that your compiler has embraced the provenance model, yes. But that does mean that you can't assume that the language pointer is the one the machine gave you and therefore probably can't write code that does anything with pointers other than pass them around as opaque handles. No testing their alignment, no comparing them to each other.
      - pjc50 2y ago
        > specifically the language is deliberately inventing things that cannot be so on hardware So what? The standard has "undefined behavior"; real implementations always do something, even if that something cannot be determined in advance. The standard is the standard, it's not a machine.
- nine_k 2y ago
  Wow, look, move semantics! An advanced compiler could even check for that...
- teo_zero 2y ago
  Would it be acceptable to save the old address as a uintptr_t before the realloc(), and compare it with the new one after?
  - saagarjha 2y ago
    You could do it legally, but the results are not required to be what you want them to be.
- matheusmoreira 2y ago
  This is only ever a problem in hosted C and its standard library malloc and free, right? If you write freestanding C with your own memory allocator you made from scratch, then the compiler will never make such assumptions. Right? That wheel is certainly in need of some serious reinvention anyway. This seems like a good starting point: https://nullprogram.com/blog/2023/12/17/ https://nullprogram.com/blog/2023/12/17/
matheusmoreira 2y ago
It doesn't mean that but maybe it actually should. The fact these perfectly good intuitions don't apply because of completely irrelevant reasons is a huge cause of friction and bugs. Your article contains the perfect example: > UB on integer overflow is a compiler-only concept > every target supported by the compiler will do the obvious thing and just produce an overflowing result Pretty much every computer that matters works the way you expect it to: the value will overflow, some flag will be set, etc. The compiler couldn't care less though. The compiler decrees that it shall be undefined. You know it works but it doesn't because the compiler refuses to do what you expect it to do because it's "undefined". Well then just define it for god's sake. I'm so tired of the uncertainty. Tired of playing these games with the compiler. Undefined? I don't even want to read that word ever again. Let's define this right now. -fwrapv There. It is now defined to be what everyone expects and wants. Now the optimizer won't be getting "clever" with this and generating complete garbage code as result. The compiler's optimizer is no longer your enemy in this case. It won't be erroneously optimizing entire loops to some random constant anymore. One would think they'd emit a warning instead in these cases but no. Strict aliasing is yet another massive pain that you need to deal with because of compilers and their "optimizations". People have actually told me to "launder" pointers through asm statements so the compiler can't make nonsense assumptions about that stuff. This is C. We do things like write memory allocators. Of course we're going to alias stuff. I have a structure type, I have a byte buffer with the data, of course I want to overlay the type on the byte buffer and just access it directly. Why is it that the compiler just doesn't let me reinterpret memory however I want? Compiler has no business making my life hell because of this. Turns out the C standard prescribes some "strict aliasing" nonsense because Fortran has it or some other reason nobody really cares about. If you're doing anything at all in C you're probably violating this. -fno-strict-aliasing There. You can now do what you want to do without the compiler getting all clever about it and ruining your day with nonsense code generation. Now it won't be reordering your code into nonsense just because it can "prove" that two pointers can't be equal even though you literally assigned one pointer to the other. Over time I've built up this little collection of C compiler flags and they've become unconditional overrides in all my makefiles. No matter what users pass in via CFLAGS or whatever, these little things still get disabled. I have no idea what the performance impact is and honestly I don't care. There's quite a lot of documented compiler flags, I couldn't evaulate every single one of them. If anyone here knows of any other useful flags that define the undefined, please reply to this comment. I'll add them to my makefiles without thinking twice.

mahkoh 2y ago

Provenance might be used as justification now but the actual rules are simpler and stricter. After freeing (or reallocing) a pointer, the application must not inspect the pointer value anymore. Even `new_pointer == old_pointer` is not allowed. IIRC, one justification for this was to account for systems with non-flat memory where inspecting the value of the old pointer might cause a processor exception.

Scaevolus 2y ago
It also makes sense for an allocator that can satisfy new allocations with recently freed allocations.
RustyRussell 2y ago
> Even `new_pointer == old_pointer` is not allowed. This is legal. But dereferencing old_pointer even after this check had passed is undefined.
- anttihaapala 2y ago
  No, it is not, and it has never been OK for as long as C has been standardized by ANSI or ISO. C89 says that "The value of a pointer that refers to freed space is indeterminate." and that behavior is undefined "upon use ... of indeterminately-valued objects", hence a compiled program can e.g. behave as if `new_pointer == old_pointer` even though the object was relocated in memory.
  - JonChesterfield 2y ago
    That standard says free takes a void*, i.e. it takes the pointer by value, and that passing values by value to a function prevents the callee from changing them. More literally, your pointer value is copied into a register before the call into libc, so free can't change the value even if it wants to. Realloc can't either for the same reason. That provenance has come up with the premise that a ! = a for a 64bit integer value is an error in the formalism, specifically the language is deliberately inventing things that cannot be so on hardware.
    - layer8 2y ago
      Note that (a) pointer representations can contain bits representing the provenance that don’t participate in equality comparison, and (b) a “C implementation” in terms of the C standard includes the compiler, which means the compiler is allowed to track pointer provenance statically even when the representation in memory doesn’t differ (e.g. exactly what GCC is doing here). In addition, a C implementation is also allowed to track pointer provenance externally at runtime, again even if the pointer representation (as exposed by memcpy into a char buffer) is the same.
      - JonChesterfield 2y ago
        To the extent that your compiler has embraced the provenance model, yes. But that does mean that you can't assume that the language pointer is the one the machine gave you and therefore probably can't write code that does anything with pointers other than pass them around as opaque handles. No testing their alignment, no comparing them to each other.
    - pjc50 2y ago
      > specifically the language is deliberately inventing things that cannot be so on hardware So what? The standard has "undefined behavior"; real implementations always do something, even if that something cannot be determined in advance. The standard is the standard, it's not a machine.
nine_k 2y ago
Wow, look, move semantics! An advanced compiler could even check for that...
teo_zero 2y ago
Would it be acceptable to save the old address as a uintptr_t before the realloc(), and compare it with the new one after?
- saagarjha 2y ago
  You could do it legally, but the results are not required to be what you want them to be.
matheusmoreira 2y ago
This is only ever a problem in hosted C and its standard library malloc and free, right? If you write freestanding C with your own memory allocator you made from scratch, then the compiler will never make such assumptions. Right? That wheel is certainly in need of some serious reinvention anyway. This seems like a good starting point: https://nullprogram.com/blog/2023/12/17/ https://nullprogram.com/blog/2023/12/17/

matheusmoreira 2y ago

It doesn't mean that but maybe it actually should. The fact these perfectly good intuitions don't apply because of completely irrelevant reasons is a huge cause of friction and bugs. Your article contains the perfect example: > UB on integer overflow is a compiler-only concept > every target supported by the compiler will do the obvious thing and just produce an overflowing result Pretty much every computer that matters works the way you expect it to: the value will overflow, some flag will be set, etc. The compiler couldn't care less though. The compiler decrees that it shall be undefined. You know it works but it doesn't because the compiler refuses to do what you expect it to do because it's "undefined". Well then just define it for god's sake. I'm so tired of the uncertainty. Tired of playing these games with the compiler. Undefined? I don't even want to read that word ever again. Let's define this right now. -fwrapv There. It is now defined to be what everyone expects and wants. Now the optimizer won't be getting "clever" with this and generating complete garbage code as result. The compiler's optimizer is no longer your enemy in this case. It won't be erroneously optimizing entire loops to some random constant anymore. One would think they'd emit a warning instead in these cases but no. Strict aliasing is yet another massive pain that you need to deal with because of compilers and their "optimizations". People have actually told me to "launder" pointers through asm statements so the compiler can't make nonsense assumptions about that stuff. This is C. We do things like write memory allocators. Of course we're going to alias stuff. I have a structure type, I have a byte buffer with the data, of course I want to overlay the type on the byte buffer and just access it directly. Why is it that the compiler just doesn't let me reinterpret memory however I want? Compiler has no business making my life hell because of this. Turns out the C standard prescribes some "strict aliasing" nonsense because Fortran has it or some other reason nobody really cares about. If you're doing anything at all in C you're probably violating this. -fno-strict-aliasing There. You can now do what you want to do without the compiler getting all clever about it and ruining your day with nonsense code generation. Now it won't be reordering your code into nonsense just because it can "prove" that two pointers can't be equal even though you literally assigned one pointer to the other. Over time I've built up this little collection of C compiler flags and they've become unconditional overrides in all my makefiles. No matter what users pass in via CFLAGS or whatever, these little things still get disabled. I have no idea what the performance impact is and honestly I don't care. There's quite a lot of documented compiler flags, I couldn't evaulate every single one of them. If anyone here knows of any other useful flags that define the undefined, please reply to this comment. I'll add them to my makefiles without thinking twice.