6 ms·

> I suspect most have a false sense of security as the compiler will elide all stores that are happening in a struct about to be freed The magic field is reset

> I suspect most have a false sense of security as the compiler will elide all stores that are happening in a struct about to be freed The magic field is reset before returning the pointer to the allocator, so it’s definitely a live write to a valid pointer.

geofft 5y ago
It's a live write to a valid pointer that's never read from, so it can't have any influence on the program behavior if the rest of the program is valid C. Compilers are free to optimize that (and they basically need to for performance reasons - unused writes are common in normal code).
- nyc_pizzadev 5y ago
  The pointer is read by the allocator, which then performs its own set of writes and management. I think you guys might be confusing stack memory and dynamic memory here. Interestingly enough, if you were to incorrectly use stack memory in this scenario, the magic checks should trigger, pointing out improper memory usage.
  - DSMan195276 5y ago
    Allocation and free functions from the standard library are 'special' though and do actually get their own logic in regards to whether a pointer is necessary or not. If you do some simple tests you can see that gcc will optimize away entire `malloc()` and `free()` calls if it can prove the resulting memory or assignments are unnecessary (Ex. Store a constant to it, and then read the constant). It's willing to do that because the behavior of those functions is defined by the standard, so it 'knows' that removing those calls has no affect on the behavior of the program. I'm pretty sure you can get similar optimization behavior when you mark functions with special attributes, I'm not 100% sure on that point though. So for the Linux Kernel I'm not sure that kind of optimization would ever be done since obviously it's not using the standard defined functions and the compiler might not be given enough information to know the functions have the same semantics as 'malloc()' and 'free()'. I personally was able to get the exact behavior described here by compiling the below code using `-O2`. The volatile write just ensures the first constant write and the malloc itself cannot be optimized out (and that does happen! Take out the volatile and there are no calls to malloc in the result), but gcc is still free to do whatever it wants with the other write and for me it's completely gone in the resulting program. int main() { int *p = malloc(sizeof(*p)); *(volatile int *)p = 20; printf("p=%d\n", *p); *p = 30; /* this is gone from the -O2 compiled code */ free(p); return 0; } The relevant part of the assembly looks like this, I don't think I'm missing anything in that the 30 assignment is completely gone: push $0x4 call 460 <malloc@plt> movl $0x14,(%eax) # Assignment of 20 mov %eax,%esi pop %eax lea -0x1930(%ebx),%eax pop %edx pushl (%esi) # push the 20 push %eax call 440 <printf@plt> mov %esi,(%esp) # load address to pass to free(), no assignment between printf() and free() call 450 <free@plt>
    - pjmlp 5y ago
      Yes, it is gone. You can play with it on Compiler Explorer. https://godbolt.org/z/Ea3WPcad1 https://godbolt.org/z/Ea3WPcad1
    - nyc_pizzadev 5y ago
      The point of the magic values are that these values are eventually checked. This should break out of these kinds of optimizations. Edit note: allocators are user space libraries (stdlib) and not part of the C spec. Use-after-free is extremely unsafe, however its completely valid C. 2nd note: running all programs below give me the expected result, writes to pointers are live, regardless if they are freed. So please provide more concrete steps to reproduce your results.
      - geofft 5y ago
        No, it should not, because it is impossible for the check to fail in valid C. A use-after-free is invalid C. Type confusion is invalid C. If you have the function void free_ws(struct ws *ptr) { ptr->magic = 0xdeadc0de; free(ptr); } there is no possible circumstance, in valid C, where ptr->magic could be read and have its value equal to 0xdeadc0de. That object is freed immediately after that write. If you do a read in some other function, say char read_from_ws(struct ws *ptr) { if (ptr->magic != WS_MAGIC) { exit(1); } return ptr->s[0]; } it is impossible for the pointer passed into read_from_ws to be a pointer that has passed through free_ws, i.e., it is impossible for ptr->magic to have been set to 0xdeadc0de by free_ws. Therefore, free_ws doesn't need to actually do the write. You're right that the correct magic check is not dead, and cannot be eliminated by the same logic. But the effect there is that the magic check always succeeds!
        vlovich123 5y ago
        If you compile with LTO the compiler will see through your obfuscation. Play dangerous games assuming the compiler is stupid, win dangerous prizes.
      - DSMan195276 5y ago
- cogman10 5y ago
  In C and C++, It's unlikely the compiler will optimize that because of pointer aliasing. It's basically impossible for the compiler to know if a given pointer will ever be re-referenced in the future due to pointer math or in an extreme case, explicit pointer addresses (happens semi frequently in the embedded world).
  - Jweb_Guru 5y ago
    You can put the code into godbolt right now and see that it's optimized out. The C and C++ standards say the compiler can make plenty of assumptions about what can and can't be aliased, and they generally won't hesitate to exploit this for optimization purposes. If they did not do this, C++ in particular would be pretty screwed as a high performance language (C might still fare okay in benchmarks, though not in real programs).
  - DSMan195276 5y ago
    If you're passing the pointer to the standard library's `free()` then the compiler doesn't actually have to worry about pointer aliasing - it's undefined behavior to use a pointer after it has been passed to `free()`, so it can conclude it never happens in your program. FWIW most of what you're referring to (reaching a free'd pointer via pointer arithmetic, or going straight to its address) is actually undefined behavior according to the C standard, what you're allowed to do with pointers is more limited than that. There are ways of getting around some of that (obviously, since it's necessary in embedded and other low-level situations), but it requires knowledge about your specific compiler and/or potentially the use of extensions or similar things not part of the standard.
- turminal 5y ago
  It can be turned of though. Compiler optimizations performed around memory allocations are weird sometimes.
  - geofft 5y ago
    Yes, but this is by far the least weird type of optimization, and the one you'd least want to turn off. It's the sort taught in undergraduate compiler design classes. Here's a piece of code that would go slower if you turned it off: for (i = 0; i < 1000; i++) { struct foobar *ptr = malloc(sizeof(struct foobar)); ptr->this = a[i]; ptr->that = b[i]; if (ptr->this > 0) { do_stuff(ptr); } free(ptr); } You only need to write to ptr->that if the condition succeeds. (And frankly you only need to do the allocation at all if a[i] > 0.) Compilers that don't do this get rejected for compilers that do.

geofft 5y ago

It's a live write to a valid pointer that's never read from, so it can't have any influence on the program behavior if the rest of the program is valid C. Compilers are free to optimize that (and they basically need to for performance reasons - unused writes are common in normal code).

nyc_pizzadev 5y ago
The pointer is read by the allocator, which then performs its own set of writes and management. I think you guys might be confusing stack memory and dynamic memory here. Interestingly enough, if you were to incorrectly use stack memory in this scenario, the magic checks should trigger, pointing out improper memory usage.
- DSMan195276 5y ago
  Allocation and free functions from the standard library are 'special' though and do actually get their own logic in regards to whether a pointer is necessary or not. If you do some simple tests you can see that gcc will optimize away entire `malloc()` and `free()` calls if it can prove the resulting memory or assignments are unnecessary (Ex. Store a constant to it, and then read the constant). It's willing to do that because the behavior of those functions is defined by the standard, so it 'knows' that removing those calls has no affect on the behavior of the program. I'm pretty sure you can get similar optimization behavior when you mark functions with special attributes, I'm not 100% sure on that point though. So for the Linux Kernel I'm not sure that kind of optimization would ever be done since obviously it's not using the standard defined functions and the compiler might not be given enough information to know the functions have the same semantics as 'malloc()' and 'free()'. I personally was able to get the exact behavior described here by compiling the below code using `-O2`. The volatile write just ensures the first constant write and the malloc itself cannot be optimized out (and that does happen! Take out the volatile and there are no calls to malloc in the result), but gcc is still free to do whatever it wants with the other write and for me it's completely gone in the resulting program. int main() { int *p = malloc(sizeof(*p)); *(volatile int *)p = 20; printf("p=%d\n", *p); *p = 30; /* this is gone from the -O2 compiled code */ free(p); return 0; } The relevant part of the assembly looks like this, I don't think I'm missing anything in that the 30 assignment is completely gone: push $0x4 call 460 <malloc@plt> movl $0x14,(%eax) # Assignment of 20 mov %eax,%esi pop %eax lea -0x1930(%ebx),%eax pop %edx pushl (%esi) # push the 20 push %eax call 440 <printf@plt> mov %esi,(%esp) # load address to pass to free(), no assignment between printf() and free() call 450 <free@plt>
  - pjmlp 5y ago
    Yes, it is gone. You can play with it on Compiler Explorer. https://godbolt.org/z/Ea3WPcad1 https://godbolt.org/z/Ea3WPcad1
  - nyc_pizzadev 5y ago
    The point of the magic values are that these values are eventually checked. This should break out of these kinds of optimizations. Edit note: allocators are user space libraries (stdlib) and not part of the C spec. Use-after-free is extremely unsafe, however its completely valid C. 2nd note: running all programs below give me the expected result, writes to pointers are live, regardless if they are freed. So please provide more concrete steps to reproduce your results.
    - geofft 5y ago
      No, it should not, because it is impossible for the check to fail in valid C. A use-after-free is invalid C. Type confusion is invalid C. If you have the function void free_ws(struct ws *ptr) { ptr->magic = 0xdeadc0de; free(ptr); } there is no possible circumstance, in valid C, where ptr->magic could be read and have its value equal to 0xdeadc0de. That object is freed immediately after that write. If you do a read in some other function, say char read_from_ws(struct ws *ptr) { if (ptr->magic != WS_MAGIC) { exit(1); } return ptr->s[0]; } it is impossible for the pointer passed into read_from_ws to be a pointer that has passed through free_ws, i.e., it is impossible for ptr->magic to have been set to 0xdeadc0de by free_ws. Therefore, free_ws doesn't need to actually do the write. You're right that the correct magic check is not dead, and cannot be eliminated by the same logic. But the effect there is that the magic check always succeeds!
      - vlovich123 5y ago
        If you compile with LTO the compiler will see through your obfuscation. Play dangerous games assuming the compiler is stupid, win dangerous prizes.
    - DSMan195276 5y ago
cogman10 5y ago
In C and C++, It's unlikely the compiler will optimize that because of pointer aliasing. It's basically impossible for the compiler to know if a given pointer will ever be re-referenced in the future due to pointer math or in an extreme case, explicit pointer addresses (happens semi frequently in the embedded world).
- Jweb_Guru 5y ago
  You can put the code into godbolt right now and see that it's optimized out. The C and C++ standards say the compiler can make plenty of assumptions about what can and can't be aliased, and they generally won't hesitate to exploit this for optimization purposes. If they did not do this, C++ in particular would be pretty screwed as a high performance language (C might still fare okay in benchmarks, though not in real programs).
- DSMan195276 5y ago
  If you're passing the pointer to the standard library's `free()` then the compiler doesn't actually have to worry about pointer aliasing - it's undefined behavior to use a pointer after it has been passed to `free()`, so it can conclude it never happens in your program. FWIW most of what you're referring to (reaching a free'd pointer via pointer arithmetic, or going straight to its address) is actually undefined behavior according to the C standard, what you're allowed to do with pointers is more limited than that. There are ways of getting around some of that (obviously, since it's necessary in embedded and other low-level situations), but it requires knowledge about your specific compiler and/or potentially the use of extensions or similar things not part of the standard.
turminal 5y ago
It can be turned of though. Compiler optimizations performed around memory allocations are weird sometimes.
- geofft 5y ago
  Yes, but this is by far the least weird type of optimization, and the one you'd least want to turn off. It's the sort taught in undergraduate compiler design classes. Here's a piece of code that would go slower if you turned it off: for (i = 0; i < 1000; i++) { struct foobar *ptr = malloc(sizeof(struct foobar)); ptr->this = a[i]; ptr->that = b[i]; if (ptr->this > 0) { do_stuff(ptr); } free(ptr); } You only need to write to ptr->that if the condition succeeds. (And frankly you only need to do the allocation at all if a[i] > 0.) Compilers that don't do this get rejected for compilers that do.