5 ms·

I wonder how long until programs start to accidentally depend on it, the same way some programs currently accidentally depend on freeing memory not immediately

by cesarb 4y ago

I wonder how long until programs start to accidentally depend on it, the same way some programs currently accidentally depend on freeing memory not immediately overwriting it. For instance, I can imagine a program which accidentally follows a dangling pointer to an already freed structure, and reads another pointer from within that structure, not crashing because it ends up checking this later pointer against NULL... until the stars align and the memory used by the freed structure has been returned to the operating system, or overwritten by a later allocation.

tlb 4y ago
On Mac, malloc has several options to help find such bugs. If you set MallocScribble=1 in the environment, it fills blocks with 0xaa on alloc and 0x55 on free. On Linux, running `env MALLOC_PERTURB_=43605 myprogram` will do the same.
- speedgoose 4y ago
  That should be the default behaviour, perhaps with random values too.
  - kergonath 4y ago
    It’s easier when it is a specified value. Easier to see in a debugger and to diagnose. Sure, it can occasionally be interpreted as meaning something, but so can random gibberish. As long as everything is overwritten, there is no privacy issue.
    - cesarb 4y ago
      > Sure, it can occasionally be interpreted as meaning something, but so can random gibberish. Some gibberish is better than others. For instance, on x86 (64-bit, 32-bit, or even 16-bit), filling with the 0xcc byte means that, if executed as code, it will trap (0xcc is the single-byte INT3 "single-step breakpoint" instruction). And for most Unix-style operating systems, which put the kernel in the high half of the address space, the filler bytes should have the high bit set, so that treating them as pointers will trap (the pointer will have its high bit set, pointing to kernel memory, which user space is forbidden from accessing).
      - wokkel 4y ago
        Or just use valgrind.
        speedgoose 4y ago
        Or just use a memory safer programming langage.
        bayindirh 4y ago
        If your use case fits into capabilities of such language, and you have the necessary libraries, and bindings, etc., then why not?
      - arcticbull 4y ago
        Having a different pre-defined value for allocation vs deallocation scribble is nice because you can then tell if the error happened pre-initialization or post-free. $AA and $55 also weren't chosen by accident one assumes, $AA is 1010_1010 and $55 is 0101_0101. Consistent values help with reproduction. The Linux setup where you can choose your own scribbles seems to meet your needs no?
      - saagarjha 4y ago
        > For instance, on x86 (64-bit, 32-bit, or even 16-bit), filling with the 0xcc byte means that, if executed as code, it will trap (0xcc is the single-byte INT3 "single-step breakpoint" instruction). Trying to execute off the heap is going to trap anyways.
  - saagarjha 4y ago
    It’s slower.
- CGamesPlay 4y ago
  For the confused, 43605 in decimal is 0xaa55 in hexadecimal.
cpeterso 4y ago
As long as Apple's allocator continues to zero freed memory, programs accidentally depending on that behavior will only be a problem for other operating systems.
- cesarb 4y ago
  > As long as Apple's allocator continues to zero freed memory, programs accidentally depending on that behavior will only be a problem for other operating systems. As I mentioned, only until that freed block is reused for a new allocation, which means it will no longer be all-zeros; or until the memory allocator decides it's time to unmap that region, which means it will trap (SIGSEGV or similar).
  - kergonath 4y ago
    > which means it will trap Is it really a problem? Failing harder and earlier is a good thing if it means the bug has more chances of being detected and fixed.
  - ncmncm 4y ago
    Memory allocators generally don't unmap anything, because the performance hit on any other threads running is severe. When memory is unmapped, caches of memory mappings of other threads are discarded, via inter-core interrupts. Then they get misses until their cache is restored. In what might be a multi-thread program, you don't fool with the memory map without very good reasons. Mapping new pages, or marking a page r/o or r/w is OK; anything else, probably not.
    - zerohp 4y ago
      > When memory is unmapped, caches of memory mappings of other threads are discarded, via inter-core interrupts. ARM architecture does not need to interrupt for TLB shootdown.
- Taniwha 4y ago
  Possibly but there's the potential reverse problems where Apple programs start depending on newly allocated memory being zero .... and those programs start failing elsewhere ...
gavinray 4y ago
"the same way some programs currently accidentally depend on freeing memory not immediately overwriting it." Maybe a dumb question, but what do you mean by this? Could you give a C/C++ pseudo-example?
- Someone 4y ago
  Not the poster, but it’s fairly easy to accidentally do the equivalent of free(f); f->close(); in code that does a few malloc, if/else’s, etc. between those calls. Stuff like that gets hard to spot in more complex code, especially in the presence of C++ copy constructors, move constructors, etc, where the compiler helpfully inserts that call to free for you. Bugs in copy constructors also can lead to double frees when the destructors of an object and its copy both call delete on a pointer that they think they exclusively own. Between those two destructor calls, you can easily get a use after free. Example: https://stackoverflow.com/a/64014035 https://stackoverflow.com/a/64014035 If the allocator doesn’t change the content of freed blocks, such bugs can go unnoticed, making the program “accidentally depend on freeing memory not immediately overwriting it”.
  - ajross 4y ago
    > especially in the presence of C++ copy constructors, move constructors, etc, You only mean the former, not both. The only reason for the existence of the giant rats' nest of complexity that is C++ move semantics is eliminating this particular mistake.
- wnissen 4y ago
  Ooh, I have a good one for this. If you have a C++ `std::string` object passing into a C API that takes a null-terminated `char *`, there's a convenience method `std::string::c_str()` to produce the C string from the C++ one. char const* inputFilename = std::string("path/to/file").c_str(); call_to_C_API(inputFilename); Boom. You can typically access `inputFilename` until the next allocation on the stack, but it will almost always go bad quickly. This isn't allocation on the heap, which usually requires another level of indirection, but you get the idea.
diebeforei485 4y ago
This is better than the status quo of use-after-free being abused for security exploits as part of an exploit chain for sandbox escape.
pjscott 4y ago
I urge anybody using a C family language to become familiar with AddressSanitizer [1], a tool that (with support from the compiler) detects an impressive variety of memory bugs at runtime. The scenario you describe would result in the program immediately crashing with a read-after-free error, along with stack traces for the most recent calls to malloc and free that involved that address. It's remarkably useful and straightforward, and Apple's tooling has good support for it, along with Xcode integration [2]. [1] https://clang.llvm.org/docs/AddressSanitizer.html https://clang.llvm.org/docs/AddressSanitizer.html [2] https://developer.apple.com/documentation/xcode/diagnosing-memory-thread-and-crash-issues-early https://developer.apple.com/documentation/xcode/diagnosing-m...
- dlundqvist 4y ago
  This. I personally think ASAN and it's friends are some of, if not the, best things to happen to C/C++ in the last decade. We use it at work, I use it in all my personal projects. We've found "countless" bugs with ASAN together with LSAN. It should be part any project from the start, run as much as possible with ASAN when the performance hit can be afforded.
  - pseudoramble 4y ago
    Just confirming, but you're taking about Leak Sanitizer? https://clang.llvm.org/docs/LeakSanitizer.html https://clang.llvm.org/docs/LeakSanitizer.html. Not really familiar with these tools, definitely not the acronyms.
    - dlundqvist 4y ago
      Sorry, yes, I meant Leak Sanitizer.
      - pseudoramble 4y ago
        No need to be sorry! I appreciate it! It was actually in the link the parent posted. I just missed it at first.

by cesarb 4y ago

tlb 4y ago

On Mac, malloc has several options to help find such bugs. If you set MallocScribble=1 in the environment, it fills blocks with 0xaa on alloc and 0x55 on free. On Linux, running `env MALLOC_PERTURB_=43605 myprogram` will do the same.

speedgoose 4y ago
That should be the default behaviour, perhaps with random values too.
- kergonath 4y ago
  It’s easier when it is a specified value. Easier to see in a debugger and to diagnose. Sure, it can occasionally be interpreted as meaning something, but so can random gibberish. As long as everything is overwritten, there is no privacy issue.
  - cesarb 4y ago
    > Sure, it can occasionally be interpreted as meaning something, but so can random gibberish. Some gibberish is better than others. For instance, on x86 (64-bit, 32-bit, or even 16-bit), filling with the 0xcc byte means that, if executed as code, it will trap (0xcc is the single-byte INT3 "single-step breakpoint" instruction). And for most Unix-style operating systems, which put the kernel in the high half of the address space, the filler bytes should have the high bit set, so that treating them as pointers will trap (the pointer will have its high bit set, pointing to kernel memory, which user space is forbidden from accessing).
    - wokkel 4y ago
      Or just use valgrind.
      - speedgoose 4y ago
        Or just use a memory safer programming langage.
        bayindirh 4y ago
        If your use case fits into capabilities of such language, and you have the necessary libraries, and bindings, etc., then why not?
    - arcticbull 4y ago
      Having a different pre-defined value for allocation vs deallocation scribble is nice because you can then tell if the error happened pre-initialization or post-free. $AA and $55 also weren't chosen by accident one assumes, $AA is 1010_1010 and $55 is 0101_0101. Consistent values help with reproduction. The Linux setup where you can choose your own scribbles seems to meet your needs no?
    - saagarjha 4y ago
      > For instance, on x86 (64-bit, 32-bit, or even 16-bit), filling with the 0xcc byte means that, if executed as code, it will trap (0xcc is the single-byte INT3 "single-step breakpoint" instruction). Trying to execute off the heap is going to trap anyways.
- saagarjha 4y ago
  It’s slower.
CGamesPlay 4y ago
For the confused, 43605 in decimal is 0xaa55 in hexadecimal.

cpeterso 4y ago

As long as Apple's allocator continues to zero freed memory, programs accidentally depending on that behavior will only be a problem for other operating systems.

cesarb 4y ago
> As long as Apple's allocator continues to zero freed memory, programs accidentally depending on that behavior will only be a problem for other operating systems. As I mentioned, only until that freed block is reused for a new allocation, which means it will no longer be all-zeros; or until the memory allocator decides it's time to unmap that region, which means it will trap (SIGSEGV or similar).
- kergonath 4y ago
  > which means it will trap Is it really a problem? Failing harder and earlier is a good thing if it means the bug has more chances of being detected and fixed.
- ncmncm 4y ago
  Memory allocators generally don't unmap anything, because the performance hit on any other threads running is severe. When memory is unmapped, caches of memory mappings of other threads are discarded, via inter-core interrupts. Then they get misses until their cache is restored. In what might be a multi-thread program, you don't fool with the memory map without very good reasons. Mapping new pages, or marking a page r/o or r/w is OK; anything else, probably not.
  - zerohp 4y ago
    > When memory is unmapped, caches of memory mappings of other threads are discarded, via inter-core interrupts. ARM architecture does not need to interrupt for TLB shootdown.
Taniwha 4y ago
Possibly but there's the potential reverse problems where Apple programs start depending on newly allocated memory being zero .... and those programs start failing elsewhere ...

gavinray 4y ago

"the same way some programs currently accidentally depend on freeing memory not immediately overwriting it." Maybe a dumb question, but what do you mean by this? Could you give a C/C++ pseudo-example?

Someone 4y ago
Not the poster, but it’s fairly easy to accidentally do the equivalent of free(f); f->close(); in code that does a few malloc, if/else’s, etc. between those calls. Stuff like that gets hard to spot in more complex code, especially in the presence of C++ copy constructors, move constructors, etc, where the compiler helpfully inserts that call to free for you. Bugs in copy constructors also can lead to double frees when the destructors of an object and its copy both call delete on a pointer that they think they exclusively own. Between those two destructor calls, you can easily get a use after free. Example: https://stackoverflow.com/a/64014035 https://stackoverflow.com/a/64014035 If the allocator doesn’t change the content of freed blocks, such bugs can go unnoticed, making the program “accidentally depend on freeing memory not immediately overwriting it”.
- ajross 4y ago
  > especially in the presence of C++ copy constructors, move constructors, etc, You only mean the former, not both. The only reason for the existence of the giant rats' nest of complexity that is C++ move semantics is eliminating this particular mistake.
wnissen 4y ago
Ooh, I have a good one for this. If you have a C++ `std::string` object passing into a C API that takes a null-terminated `char *`, there's a convenience method `std::string::c_str()` to produce the C string from the C++ one. char const* inputFilename = std::string("path/to/file").c_str(); call_to_C_API(inputFilename); Boom. You can typically access `inputFilename` until the next allocation on the stack, but it will almost always go bad quickly. This isn't allocation on the heap, which usually requires another level of indirection, but you get the idea.

diebeforei485 4y ago

This is better than the status quo of use-after-free being abused for security exploits as part of an exploit chain for sandbox escape.

pjscott 4y ago

I urge anybody using a C family language to become familiar with AddressSanitizer [1], a tool that (with support from the compiler) detects an impressive variety of memory bugs at runtime. The scenario you describe would result in the program immediately crashing with a read-after-free error, along with stack traces for the most recent calls to malloc and free that involved that address. It's remarkably useful and straightforward, and Apple's tooling has good support for it, along with Xcode integration [2]. [1] https://clang.llvm.org/docs/AddressSanitizer.html https://clang.llvm.org/docs/AddressSanitizer.html [2] https://developer.apple.com/documentation/xcode/diagnosing-memory-thread-and-crash-issues-early https://developer.apple.com/documentation/xcode/diagnosing-m...

dlundqvist 4y ago
This. I personally think ASAN and it's friends are some of, if not the, best things to happen to C/C++ in the last decade. We use it at work, I use it in all my personal projects. We've found "countless" bugs with ASAN together with LSAN. It should be part any project from the start, run as much as possible with ASAN when the performance hit can be afforded.
- pseudoramble 4y ago
  Just confirming, but you're taking about Leak Sanitizer? https://clang.llvm.org/docs/LeakSanitizer.html https://clang.llvm.org/docs/LeakSanitizer.html. Not really familiar with these tools, definitely not the acronyms.
  - dlundqvist 4y ago
    Sorry, yes, I meant Leak Sanitizer.
    - pseudoramble 4y ago
      No need to be sorry! I appreciate it! It was actually in the link the parent posted. I just missed it at first.