5 ms·

What is the use case for valgrind vs memory sanitizer?

pengaru 3y ago
valgrind does far more than asan-like analysis. --tool=callgrind for instance
mtklein 3y ago
Valgrind is like all the sanitizers (memory, thread, leak, address, undefined) all together, plus performance and memory profilers, all for uninstrumented code. For me the real key is the last part. If you're happy building and testing with sanitizers from source, I don't think there's much more to gain by adding Valgrind to the mix, but if sanitizers aren't available or you find yourself with a binary and not source, I'd check out Valgrind.
- nlewycky 3y ago
  > Valgrind is like all the sanitizers (memory, thread, leak, address, undefined) all together, plus performance and memory profilers, all for uninstrumented code. Valgrind as memcheck can do memory and leak and partially do address sanitizer, and when run as helgrind can do thread sanitizer. Valgrind doesn't do -fsanitize=undefined. Undefined -- UBSan -- in particular doesn't overlap with valgrind. Your CPU uses the same instructions for signed and unsigned integer addition (thanks to 2's complement) but UBSan will catch signed integer overflow. UBSan will catch misaligned pointers whether the CPU permits unaligned accesses or not. UBSan will catch integer and float divide by zero, which valgrind ignores because at the CPU level those have defined behaviours (of setting an error flag or whatnot). As for partially doing address sanitizer, ASan improves its probability of catching invalid pointer arithmetic by intentionally spacing out stack allocations and global variables in a way that the compiler is allowed to do but which valgrind can't because valgrind has to emulate the CPU instructions as they lie. ASan also replaces malloc with its own version that attempts to give different addresses as much as possible so that the use of a freed pointer is maximally likely to not accidentally point to memory which has since been reallocated. Optionally ASan also includes special features like "after running the inliner, malloc a fresh stack for every function call" so that you can really catch stack use after return https://github.com/google/sanitizers/wiki/AddressSanitizerUseAfterReturn https://github.com/google/sanitizers/wiki/AddressSanitizerUs... I'm not deeply familiar with either ThreadSanitizer or Helgrind. The TSan devs claim that Helgrind either runs in "catches too few bugs" mode or "has too many false positives" mode and that they got it just right in TSan. I can't evaluate that claim. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35604.pdf https://static.googleusercontent.com/media/research.google.c...
JonChesterfield 3y ago
Valgrind works on arbitrary code. Even if it's doing things like jit compilation or self modifying code. Without requiring the compiler to have injected instrumentation ahead of time. There's a lot of C I've written which is only correct because I ran the test suite under valgrind while implementing it.
ahartmetz 3y ago
It can detect more errors and you don't need to recompile anything.
- hannob 3y ago
  Generally asan+msan detect more errors, and some quite significant ones. For design reasons, valgrind cannot detect most stack oob bugs. The only issue with msan is that it's a bit trickier to use, particularly if you have lots of dependencies. Both approaches have their right to exist, but I have been preaching to people that valgrind does not find all memory safety bugs that modern tools can find, and they should be testing with sanitizers. I still experience that many people are not aware of asan+co and only know valgrind.
  - dezgeg 3y ago
    How well does ASAN cope when the libraries your application uses aren't compiled with ASAN? Does it cause false negatives or false positives?
    - nlewycky 3y ago
      ASan will have false negatives on the memory accesses performed in the code which isn't compiled with ASan. It works fine, otherwise. FWIW, UBSan (-fsanitize=undefined) follows the same rule. For this reason it's common to combine them and build once with -fsanitize=address,undefined. Don't forget to pass that while linking too. ThreadSanitizer requires that all code with atomic accesses be instrumented. MemorySanitizer requires that all code which writes to a byte in memory be instrumented.
rwmj 3y ago
Valgrind is super-practical because you can use it on any existing binary. Basically every project I'm involved in has a "make check-valgrind" rule which runs all the unit tests again, but under valgrind. It has caught countless memory leaks (mainly) and memory errors. We also run it on Rust, OCaml and Perl code.
- unmole 3y ago
  > Basically every project I'm involved in has a "make check-valgrind" rule This the exact scenario where address sanitizer makes more sense. I stopped using Valgrind because ASan catches more issues, doesn't report false positives and is much much faster.
  - rwmj 3y ago
    Except you have to recompile everything so you're not testing the same binaries that you're going to ship. That's exactly the reason to use valgrind.
    - unmole 3y ago
      Having separate debug and release targets is hardly new.
      - rwmj 3y ago
        And is still a terrible idea. I build everything with debugging symbols (albeit in a separate debuginfo downloadable), and debug messages can be turned on at any time. Saved my bacon countless times in production environments.
- er4hn 3y ago
  What does it catch under Rust? Unsafe code? I'm under the impression the borrow checker would prevent the class of problems valgrind is made for.
  - 3y ago

valgrind does far more than asan-like analysis. --tool=callgrind for instance

Valgrind is like all the sanitizers (memory, thread, leak, address, undefined) all together, plus performance and memory profilers, all for uninstrumented code. For me the real key is the last part. If you're happy building and testing with sanitizers from source, I don't think there's much more to gain by adding Valgrind to the mix, but if sanitizers aren't available or you find yourself with a binary and not source, I'd check out Valgrind.

nlewycky 3y ago
> Valgrind is like all the sanitizers (memory, thread, leak, address, undefined) all together, plus performance and memory profilers, all for uninstrumented code. Valgrind as memcheck can do memory and leak and partially do address sanitizer, and when run as helgrind can do thread sanitizer. Valgrind doesn't do -fsanitize=undefined. Undefined -- UBSan -- in particular doesn't overlap with valgrind. Your CPU uses the same instructions for signed and unsigned integer addition (thanks to 2's complement) but UBSan will catch signed integer overflow. UBSan will catch misaligned pointers whether the CPU permits unaligned accesses or not. UBSan will catch integer and float divide by zero, which valgrind ignores because at the CPU level those have defined behaviours (of setting an error flag or whatnot). As for partially doing address sanitizer, ASan improves its probability of catching invalid pointer arithmetic by intentionally spacing out stack allocations and global variables in a way that the compiler is allowed to do but which valgrind can't because valgrind has to emulate the CPU instructions as they lie. ASan also replaces malloc with its own version that attempts to give different addresses as much as possible so that the use of a freed pointer is maximally likely to not accidentally point to memory which has since been reallocated. Optionally ASan also includes special features like "after running the inliner, malloc a fresh stack for every function call" so that you can really catch stack use after return https://github.com/google/sanitizers/wiki/AddressSanitizerUseAfterReturn https://github.com/google/sanitizers/wiki/AddressSanitizerUs... I'm not deeply familiar with either ThreadSanitizer or Helgrind. The TSan devs claim that Helgrind either runs in "catches too few bugs" mode or "has too many false positives" mode and that they got it just right in TSan. I can't evaluate that claim. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35604.pdf https://static.googleusercontent.com/media/research.google.c...

JonChesterfield 3y ago

Valgrind works on arbitrary code. Even if it's doing things like jit compilation or self modifying code. Without requiring the compiler to have injected instrumentation ahead of time. There's a lot of C I've written which is only correct because I ran the test suite under valgrind while implementing it.

ahartmetz 3y ago

It can detect more errors and you don't need to recompile anything.

hannob 3y ago
Generally asan+msan detect more errors, and some quite significant ones. For design reasons, valgrind cannot detect most stack oob bugs. The only issue with msan is that it's a bit trickier to use, particularly if you have lots of dependencies. Both approaches have their right to exist, but I have been preaching to people that valgrind does not find all memory safety bugs that modern tools can find, and they should be testing with sanitizers. I still experience that many people are not aware of asan+co and only know valgrind.
- dezgeg 3y ago
  How well does ASAN cope when the libraries your application uses aren't compiled with ASAN? Does it cause false negatives or false positives?
  - nlewycky 3y ago
    ASan will have false negatives on the memory accesses performed in the code which isn't compiled with ASan. It works fine, otherwise. FWIW, UBSan (-fsanitize=undefined) follows the same rule. For this reason it's common to combine them and build once with -fsanitize=address,undefined. Don't forget to pass that while linking too. ThreadSanitizer requires that all code with atomic accesses be instrumented. MemorySanitizer requires that all code which writes to a byte in memory be instrumented.

rwmj 3y ago

Valgrind is super-practical because you can use it on any existing binary. Basically every project I'm involved in has a "make check-valgrind" rule which runs all the unit tests again, but under valgrind. It has caught countless memory leaks (mainly) and memory errors. We also run it on Rust, OCaml and Perl code.

unmole 3y ago
> Basically every project I'm involved in has a "make check-valgrind" rule This the exact scenario where address sanitizer makes more sense. I stopped using Valgrind because ASan catches more issues, doesn't report false positives and is much much faster.
- rwmj 3y ago
  Except you have to recompile everything so you're not testing the same binaries that you're going to ship. That's exactly the reason to use valgrind.
  - unmole 3y ago
    Having separate debug and release targets is hardly new.
    - rwmj 3y ago
      And is still a terrible idea. I build everything with debugging symbols (albeit in a separate debuginfo downloadable), and debug messages can be turned on at any time. Saved my bacon countless times in production environments.
er4hn 3y ago
What does it catch under Rust? Unsafe code? I'm under the impression the borrow checker would prevent the class of problems valgrind is made for.
- 3y ago