4 ms·

The distinction between "sanitizers" and "Valgrind-like" drawn here is quite arbitrary. I think the core distinction being drawn is that "sanitizers" use compil

by roca 7y ago

The distinction between "sanitizers" and "Valgrind-like" drawn here is quite arbitrary. I think the core distinction being drawn is that "sanitizers" use compiler-inserted instrumentation and "Valgrind-like" tools use dynamic binary translation, though the author doesn't make this clear. The conclusions drawn are questionable. The author states that "sanitizers" are fast and "Valgrind-like" tools are slow, but it actually depends on the tool. Some sanitizers are much faster than others. Some dynamic binary translation tools (e.g. DynamoRio's Dr Memory) are much faster than Valgrind. It wouldn't be hard to come up with examples where a compiler-based sanitizer is slower than a dynamic binary translation tool. It all depends on what the tool is actually checking. Items 3 and 4 are also questionable. It's certainly possible to configure dynamic binary translation tools with customized function-specific behavior. Dynamic translation tools are generally even easier to use than sanitizers because you don't have to modify your build at all, you can just run tests under some kind of wrapper command.

jonstewart 7y ago
He doesn't use the term "Valgrind-like". He only talks about Valgrind, which he calls a "great tool" (he's right!). But he is evangelizing sanitizers, in clang and gcc, which are relatively new. And, yes, they do work by adding instrumentation at compile-time, which allows them to be much faster than Valgrind (which does all its analysis at run-time). See: http://btorpey.github.io/blog/2014/03/27/using-clangs-address-sanitizer/ http://btorpey.github.io/blog/2014/03/27/using-clangs-addres... Sanitizers are great tools, too. Maybe you can quibble about his item 3, but he's dead on with item 4. If you have tests, you add a couple compiler flags, and then you'll find memory problems automatically with your unit tests (without much slowdown).
- inetknght 7y ago
  I currently work in the DNA analysis industry. We have some pretty massive data sets and computational challenges. We use C++ for the heavy lifting. There's no way we'd be able to valgrind anything but toy input (valgrind's performance cost is prohibitive) -- and there's so many different possible inputs and code paths that it'd be effectively impossible to automatically generate all possible toy inputs. So we use sanitizers, as many as we can. They're a game changer in correctness validation. I'd love for fuzzing to be a thing, but even that seems impractical on a large scale (please prove me wrong!) since it's a lot more work than just setting a compile target in cmake and ensuring that sanitizer flags are set for that compile target.
  - touisteur 7y ago
    The right approach to start fuzzing with AFL, to me, is to recycle existing unit tests or integration tests. In fact if you already have tests that create a record (sorry, struct or class instance), inject them in your part of your application, you're almost. Read your input data from a file and 'there' you go... Paralellizing or distributing an AFL session on dozens of machines is a matter of rsyncing some files. Sanitizers are also very useful there in C/C++ languages because they're mostly runtime checks. In Ada you had them already for a long time, and you can be very through with your checks (see for example validity checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.html https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.ht... , runtime stack size checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Checking.html#f4 https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Check... , and the very very powerful Initialize_Scalars that has saved me many times https://www.adacore.com/papers/exposing-uninitialized-variables-strengthening-and-extending-run-time-check https://www.adacore.com/papers/exposing-uninitialized-variab... ). Shameless plug, but if anyone is interested in an introduction to fuzzing (in Ada) I wrote something some time ago that could be useful : https://blog.adacore.com/running-american-fuzzy-lop-on-your-ada-code https://blog.adacore.com/running-american-fuzzy-lop-on-your-... . Fuzzing is fun !
    - inetknght 7y ago
      I did a little bit of fuzzing and in general I agree with you about the right approach being to build off of existing unit/integration tests. But in practical terms, that's still a lot more than "add this compiler flag" or "swap out this library".

by roca 7y ago

jonstewart 7y ago

He doesn't use the term "Valgrind-like". He only talks about Valgrind, which he calls a "great tool" (he's right!). But he is evangelizing sanitizers, in clang and gcc, which are relatively new. And, yes, they do work by adding instrumentation at compile-time, which allows them to be much faster than Valgrind (which does all its analysis at run-time). See: http://btorpey.github.io/blog/2014/03/27/using-clangs-address-sanitizer/ http://btorpey.github.io/blog/2014/03/27/using-clangs-addres... Sanitizers are great tools, too. Maybe you can quibble about his item 3, but he's dead on with item 4. If you have tests, you add a couple compiler flags, and then you'll find memory problems automatically with your unit tests (without much slowdown).

inetknght 7y ago
I currently work in the DNA analysis industry. We have some pretty massive data sets and computational challenges. We use C++ for the heavy lifting. There's no way we'd be able to valgrind anything but toy input (valgrind's performance cost is prohibitive) -- and there's so many different possible inputs and code paths that it'd be effectively impossible to automatically generate all possible toy inputs. So we use sanitizers, as many as we can. They're a game changer in correctness validation. I'd love for fuzzing to be a thing, but even that seems impractical on a large scale (please prove me wrong!) since it's a lot more work than just setting a compile target in cmake and ensuring that sanitizer flags are set for that compile target.
- touisteur 7y ago
  The right approach to start fuzzing with AFL, to me, is to recycle existing unit tests or integration tests. In fact if you already have tests that create a record (sorry, struct or class instance), inject them in your part of your application, you're almost. Read your input data from a file and 'there' you go... Paralellizing or distributing an AFL session on dozens of machines is a matter of rsyncing some files. Sanitizers are also very useful there in C/C++ languages because they're mostly runtime checks. In Ada you had them already for a long time, and you can be very through with your checks (see for example validity checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.html https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.ht... , runtime stack size checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Checking.html#f4 https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Check... , and the very very powerful Initialize_Scalars that has saved me many times https://www.adacore.com/papers/exposing-uninitialized-variables-strengthening-and-extending-run-time-check https://www.adacore.com/papers/exposing-uninitialized-variab... ). Shameless plug, but if anyone is interested in an introduction to fuzzing (in Ada) I wrote something some time ago that could be useful : https://blog.adacore.com/running-american-fuzzy-lop-on-your-ada-code https://blog.adacore.com/running-american-fuzzy-lop-on-your-... . Fuzzing is fun !
  - inetknght 7y ago
    I did a little bit of fuzzing and in general I agree with you about the right approach being to build off of existing unit/integration tests. But in practical terms, that's still a lot more than "add this compiler flag" or "swap out this library".