9 ms·

A handful of C projects I have seen use magic numbers in allocated structs to prevent use-after-free and other memory bugs[0]. Basically, in this case, when the

A handful of C projects I have seen use magic numbers in allocated structs to prevent use-after-free and other memory bugs[0]. Basically, in this case, when the ref count hits zero and the struct is freed, the magic is zeroed and any further access will be stopped. The author makes no reference of this, so I guess this isn’t a widespread safety pattern? [0] https://github.com/varnishcache/varnish-cache/blob/4ae73a5b19ec1fc87cc018c01757535667e33341/bin/varnishd/cache/cache.h#L124-L125 https://github.com/varnishcache/varnish-cache/blob/4ae73a5b1...

tjalfi 5y ago
Multics and Stratus VOS are two operating systems that use structure marking[0]. [0] https://multicians.org/thvv/marking.html https://multicians.org/thvv/marking.html
vlovich123 5y ago
It’s possible some projects do this correctly but I suspect most have a false sense of security as the compiler will elide all stores that are happening in a struct about to be freed and there’s no C/C++/LLVM language that’s really immune from this [1]. Usually a more thorough approach is to turn on malloc scribbling, ASAN or valgrind which is something Darwin’s allocator can be told to do (it’ll scribble separate uninitialized and freed patterns). I could see the appeal of there being a magic value though. I think that’s what memset_s is for so hopefully your favorite project is doing that properly. [1] http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buffer.html http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buf...
- nyc_pizzadev 5y ago
  > I suspect most have a false sense of security as the compiler will elide all stores that are happening in a struct about to be freed The magic field is reset before returning the pointer to the allocator, so it’s definitely a live write to a valid pointer.
  - geofft 5y ago
    It's a live write to a valid pointer that's never read from, so it can't have any influence on the program behavior if the rest of the program is valid C. Compilers are free to optimize that (and they basically need to for performance reasons - unused writes are common in normal code).
    - nyc_pizzadev 5y ago
      The pointer is read by the allocator, which then performs its own set of writes and management. I think you guys might be confusing stack memory and dynamic memory here. Interestingly enough, if you were to incorrectly use stack memory in this scenario, the magic checks should trigger, pointing out improper memory usage.
      - DSMan195276 5y ago
        Allocation and free functions from the standard library are 'special' though and do actually get their own logic in regards to whether a pointer is necessary or not. If you do some simple tests you can see that gcc will optimize away entire `malloc()` and `free()` calls if it can prove the resulting memory or assignments are unnecessary (Ex. Store a constant to it, and then read the constant). It's willing to do that because the behavior of those functions is defined by the standard, so it 'knows' that removing those calls has no affect on the behavior of the program. I'm pretty sure you can get similar optimization behavior when you mark functions with special attributes, I'm not 100% sure on that point though. So for the Linux Kernel I'm not sure that kind of optimization would ever be done since obviously it's not using the standard defined functions and the compiler might not be given enough information to know the functions have the same semantics as 'malloc()' and 'free()'. I personally was able to get the exact behavior described here by compiling the below code using `-O2`. The volatile write just ensures the first constant write and the malloc itself cannot be optimized out (and that does happen! Take out the volatile and there are no calls to malloc in the result), but gcc is still free to do whatever it wants with the other write and for me it's completely gone in the resulting program. int main() { int *p = malloc(sizeof(*p)); *(volatile int *)p = 20; printf("p=%d\n", *p); *p = 30; /* this is gone from the -O2 compiled code */ free(p); return 0; } The relevant part of the assembly looks like this, I don't think I'm missing anything in that the 30 assignment is completely gone: push $0x4 call 460 <malloc@plt> movl $0x14,(%eax) # Assignment of 20 mov %eax,%esi pop %eax lea -0x1930(%ebx),%eax pop %edx pushl (%esi) # push the 20 push %eax call 440 <printf@plt> mov %esi,(%esp) # load address to pass to free(), no assignment between printf() and free() call 450 <free@plt>
geofft 5y ago
This only protects you against unintentional use-after-free. If a use-after-free of struct ws is a thing you're worried about an attacker intentionally causing, in order for this to be useful, the attacker needs to control one of those four char * pointers and point them somewhere useful. Typically they'd do that by inducing the program to re-allocating the freed memory with a buffer under their control (like input from the network) and then filling it in with specific bytes. If they can do that, they can very easily fill in the magic numbers too. It's even easier than pointers because it doesn't require inferring information about the running program - the magic number is the same across all instances of Varnish and right there in the source. "Heap spray" attacks are a generalization of this where the attacker doesn't have precise enough control about what happens between the unwanted free and the reuse, but they can allocate a very large buffer (e.g., send a lot of data in from the network, or open a lot of connections) and put their data in that way. This approach would be basically perfect for defeating the "magic number" approach. (The blog post itself has a discussion of a number of more advanced variants on the "magic number" approach - see the mention of "tagging pointers with a small number of bits that are checked against object metadata on access".)
- tjalfi 5y ago
  The Stratus recommendations for structure marking would be a bit harder to defeat. Here's an excerpt from an old Stratus presentation[0] on writing robust software. Add TYPE, SIZE, VERSION, and OWNER to data structure TYPE: Unique number for each different structure SIZE: in bytes VERSION: Changed whenever structure declaration changes OWNER: Unique ID of owner, must be independent of structure contents; can be UID, least significant bits of clock, etc. [0] https://web.archive.org/web/20170303065858/http://ftp.stratus.com/vos/doc/papers/RobustProgramming.ps https://web.archive.org/web/20170303065858/http://ftp.stratu...
- nyc_pizzadev 5y ago
  > they can very easily fill in the magic numbers too Right, recreating the magic does side step this defense. The context for software security these days is defense in depth and not something like “total defense” anymore. In this case, the use of magics is more of a dev testing mechanism than a runtime protect, although it does provide great runtime protection. What this means is if you use magics with proper testing and load testing, errors should surface before you release.
pmarreck 5y ago
Solutions like this depend on the will, skill and ethics of the coder. Better IMHO to design a language in such a way that dangerous errors like this are completely impossible. (I mean... this is basically why I switched from Ruby to Elixir for web dev, eliminating an entire class of bugs... If the language itself doesn't provide an error-reduction feature, then you are reliant on other developers to "do the right thing" and lose any guarantees)

tjalfi 5y ago

Multics and Stratus VOS are two operating systems that use structure marking[0]. [0] https://multicians.org/thvv/marking.html https://multicians.org/thvv/marking.html

vlovich123 5y ago

It’s possible some projects do this correctly but I suspect most have a false sense of security as the compiler will elide all stores that are happening in a struct about to be freed and there’s no C/C++/LLVM language that’s really immune from this [1]. Usually a more thorough approach is to turn on malloc scribbling, ASAN or valgrind which is something Darwin’s allocator can be told to do (it’ll scribble separate uninitialized and freed patterns). I could see the appeal of there being a magic value though. I think that’s what memset_s is for so hopefully your favorite project is doing that properly. [1] http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buffer.html http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buf...

nyc_pizzadev 5y ago
> I suspect most have a false sense of security as the compiler will elide all stores that are happening in a struct about to be freed The magic field is reset before returning the pointer to the allocator, so it’s definitely a live write to a valid pointer.
- geofft 5y ago
  It's a live write to a valid pointer that's never read from, so it can't have any influence on the program behavior if the rest of the program is valid C. Compilers are free to optimize that (and they basically need to for performance reasons - unused writes are common in normal code).
  - nyc_pizzadev 5y ago
    The pointer is read by the allocator, which then performs its own set of writes and management. I think you guys might be confusing stack memory and dynamic memory here. Interestingly enough, if you were to incorrectly use stack memory in this scenario, the magic checks should trigger, pointing out improper memory usage.
    - DSMan195276 5y ago
      Allocation and free functions from the standard library are 'special' though and do actually get their own logic in regards to whether a pointer is necessary or not. If you do some simple tests you can see that gcc will optimize away entire `malloc()` and `free()` calls if it can prove the resulting memory or assignments are unnecessary (Ex. Store a constant to it, and then read the constant). It's willing to do that because the behavior of those functions is defined by the standard, so it 'knows' that removing those calls has no affect on the behavior of the program. I'm pretty sure you can get similar optimization behavior when you mark functions with special attributes, I'm not 100% sure on that point though. So for the Linux Kernel I'm not sure that kind of optimization would ever be done since obviously it's not using the standard defined functions and the compiler might not be given enough information to know the functions have the same semantics as 'malloc()' and 'free()'. I personally was able to get the exact behavior described here by compiling the below code using `-O2`. The volatile write just ensures the first constant write and the malloc itself cannot be optimized out (and that does happen! Take out the volatile and there are no calls to malloc in the result), but gcc is still free to do whatever it wants with the other write and for me it's completely gone in the resulting program. int main() { int *p = malloc(sizeof(*p)); *(volatile int *)p = 20; printf("p=%d\n", *p); *p = 30; /* this is gone from the -O2 compiled code */ free(p); return 0; } The relevant part of the assembly looks like this, I don't think I'm missing anything in that the 30 assignment is completely gone: push $0x4 call 460 <malloc@plt> movl $0x14,(%eax) # Assignment of 20 mov %eax,%esi pop %eax lea -0x1930(%ebx),%eax pop %edx pushl (%esi) # push the 20 push %eax call 440 <printf@plt> mov %esi,(%esp) # load address to pass to free(), no assignment between printf() and free() call 450 <free@plt>

geofft 5y ago

This only protects you against unintentional use-after-free. If a use-after-free of struct ws is a thing you're worried about an attacker intentionally causing, in order for this to be useful, the attacker needs to control one of those four char * pointers and point them somewhere useful. Typically they'd do that by inducing the program to re-allocating the freed memory with a buffer under their control (like input from the network) and then filling it in with specific bytes. If they can do that, they can very easily fill in the magic numbers too. It's even easier than pointers because it doesn't require inferring information about the running program - the magic number is the same across all instances of Varnish and right there in the source. "Heap spray" attacks are a generalization of this where the attacker doesn't have precise enough control about what happens between the unwanted free and the reuse, but they can allocate a very large buffer (e.g., send a lot of data in from the network, or open a lot of connections) and put their data in that way. This approach would be basically perfect for defeating the "magic number" approach. (The blog post itself has a discussion of a number of more advanced variants on the "magic number" approach - see the mention of "tagging pointers with a small number of bits that are checked against object metadata on access".)

tjalfi 5y ago
The Stratus recommendations for structure marking would be a bit harder to defeat. Here's an excerpt from an old Stratus presentation[0] on writing robust software. Add TYPE, SIZE, VERSION, and OWNER to data structure TYPE: Unique number for each different structure SIZE: in bytes VERSION: Changed whenever structure declaration changes OWNER: Unique ID of owner, must be independent of structure contents; can be UID, least significant bits of clock, etc. [0] https://web.archive.org/web/20170303065858/http://ftp.stratus.com/vos/doc/papers/RobustProgramming.ps https://web.archive.org/web/20170303065858/http://ftp.stratu...
nyc_pizzadev 5y ago
> they can very easily fill in the magic numbers too Right, recreating the magic does side step this defense. The context for software security these days is defense in depth and not something like “total defense” anymore. In this case, the use of magics is more of a dev testing mechanism than a runtime protect, although it does provide great runtime protection. What this means is if you use magics with proper testing and load testing, errors should surface before you release.

pmarreck 5y ago

Solutions like this depend on the will, skill and ethics of the coder. Better IMHO to design a language in such a way that dangerous errors like this are completely impossible. (I mean... this is basically why I switched from Ruby to Elixir for web dev, eliminating an entire class of bugs... If the language itself doesn't provide an error-reduction feature, then you are reliant on other developers to "do the right thing" and lose any guarantees)