11 ms·
Building better software with better tools: sanitizers vs. Valgrind
- roca 8y agoThe distinction between "sanitizers" and "Valgrind-like" drawn here is quite arbitrary. I think the core distinction being drawn is that "sanitizers" use compiler-inserted instrumentation and "Valgrind-like" tools use dynamic binary translation, though the author doesn't make this clear. The conclusions drawn are questionable. The author states that "sanitizers" are fast and "Valgrind-like" tools are slow, but it actually depends on the tool. Some sanitizers are much faster than others. Some dynamic binary translation tools (e.g. DynamoRio's Dr Memory) are much faster than Valgrind. It wouldn't be hard to come up with examples where a compiler-based sanitizer is slower than a dynamic binary translation tool. It all depends on what the tool is actually checking. Items 3 and 4 are also questionable. It's certainly possible to configure dynamic binary translation tools with customized function-specific behavior. Dynamic translation tools are generally even easier to use than sanitizers because you don't have to modify your build at all, you can just run tests under some kind of wrapper command.
- jonstewart 8y agoHe doesn't use the term "Valgrind-like". He only talks about Valgrind, which he calls a "great tool" (he's right!). But he is evangelizing sanitizers, in clang and gcc, which are relatively new. And, yes, they do work by adding instrumentation at compile-time, which allows them to be much faster than Valgrind (which does all its analysis at run-time). See: http://btorpey.github.io/blog/2014/03/27/using-clangs-address-sanitizer/ http://btorpey.github.io/blog/2014/03/27/using-clangs-addres... Sanitizers are great tools, too. Maybe you can quibble about his item 3, but he's dead on with item 4. If you have tests, you add a couple compiler flags, and then you'll find memory problems automatically with your unit tests (without much slowdown).
- inetknght 8y agoI currently work in the DNA analysis industry. We have some pretty massive data sets and computational challenges. We use C++ for the heavy lifting. There's no way we'd be able to valgrind anything but toy input (valgrind's performance cost is prohibitive) -- and there's so many different possible inputs and code paths that it'd be effectively impossible to automatically generate all possible toy inputs. So we use sanitizers, as many as we can. They're a game changer in correctness validation. I'd love for fuzzing to be a thing, but even that seems impractical on a large scale (please prove me wrong!) since it's a lot more work than just setting a compile target in cmake and ensuring that sanitizer flags are set for that compile target.
- touisteur 8y agoThe right approach to start fuzzing with AFL, to me, is to recycle existing unit tests or integration tests. In fact if you already have tests that create a record (sorry, struct or class instance), inject them in your part of your application, you're almost. Read your input data from a file and 'there' you go... Paralellizing or distributing an AFL session on dozens of machines is a matter of rsyncing some files. Sanitizers are also very useful there in C/C++ languages because they're mostly runtime checks. In Ada you had them already for a long time, and you can be very through with your checks (see for example validity checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.html https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-Checking.ht... , runtime stack size checks https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Checking.html#f4 https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-Check... , and the very very powerful Initialize_Scalars that has saved me many times https://www.adacore.com/papers/exposing-uninitialized-variables-strengthening-and-extending-run-time-check https://www.adacore.com/papers/exposing-uninitialized-variab... ). Shameless plug, but if anyone is interested in an introduction to fuzzing (in Ada) I wrote something some time ago that could be useful : https://blog.adacore.com/running-american-fuzzy-lop-on-your-ada-code https://blog.adacore.com/running-american-fuzzy-lop-on-your-... . Fuzzing is fun !
- inetknght 8y agoI did a little bit of fuzzing and in general I agree with you about the right approach being to build off of existing unit/integration tests. But in practical terms, that's still a lot more than "add this compiler flag" or "swap out this library".
- adrianN 8y agoSanitizers and Valgrind are nice, but you need a comprehensive test suite to really benefit from them. They only detect problems that are triggered by your testsuite. This is a far cry from statically proving that your software is free from e.g. use-after-free mistakes.
- jonstewart 8y agoI mean, if you don’t have tests, write some. I know that sounds glib when confronted with a horrible old big ball of mud, but it’s probably the best place to start.
- jzwinck 8y agoNo test suite, no problem. Just run in a production staging area or deploy to beta users.
- aflag 8y agoThat's better than nothing, but it would still be no proof that the are no issues, as uncommon behaviour/access patterns could take some time to appear (or will only show up after other services are introduced).
- inetknght 8y ago> or deploy to beta users. And for God's sake: your users are NOT beta testers!
- hyperman1 8y agoCan someone explain how windbg is valgrind- like? I always considered it a debugger.
- nn3 8y agowindbg has many powerful plugins, and some do automatic verification (or talk to tools that do so), so it can be a tool kit for verifiers like valgrind is.
- hyperman1 8y agoInterestibg. Are there advanced blogs or so out there, demonstrating in the real world? Raymond chen seems to only touch it's surface.
- sharkbot 8y agoThe author might be referring to the recommendation to run a program that's instrumented with Application Verifier under a debugger, which WinDbg is one of many.
- wyldfire 8y ago> Sanitizers are a game changer, they bring C++ closer to languages like Java and Rust. This is a stretch, IMO. I suppose it is ever-so-slightly-closer. > They do not bring the safety to production, since you probably do not want to use sanitizers in production or release You're right, they're not intended for use in production -- until now! ARM dedicated some virtual address bits to tagging memory [1][2][3] for ARMv8 so that we can have sanitizers in hardware. Imagine: chrome on your smartphone being protected! [1] https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile https://developer.arm.com/docs/ddi0487/latest/arm-architectu... [2] https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html https://clang.llvm.org/docs/HardwareAssistedAddressSanitizer... [3] https://arxiv.org/abs/1802.09517 https://arxiv.org/abs/1802.09517
- wyldfire 8y agoNote that valgrind can also detect uninitialized reads (or conditional considered uninitialized memory). This is possible with MSan but MSan is a little more challenging to use than ASan+UBSan or TSan. In the years that I've used the sanitizers, I have yet to discover a false positive. It's always been a real bug.
- carlmr 8y agoI tried getting MSan to run but it never links properly for me.
- deng 8y agoWhy "versus"? You use both, of course. And additionally, you use static analyzers like clang-tidy, cppcheck, PVS, you-name-it. And you're still gonna be miles away from Java and Rust w.r.t. safety. Undefined behavior is the bane of the C-language family.
- scottlamb 8y ago> Sanitizers ... do not bring the safety to production, since you probably do not want to use sanitizers in production or release My team runs ASAN in production. We run lots of servers, and we have load balancing that keeps CPU utilization equal across servers. ASAN binaries are slower (maybe 1/2 throughput / 2X "local" cpu-bound latency?) but not so much slower that running it on a small fraction of our servers means we can't meet our SLO. And it's been a huge help to us. Once in a while, we have some segfault that's pretty mysterious otherwise, as the stack trace where memory corruption is discovered can be totally unrelated to the stack trace where the problem was caused. If it happens on the ASAN servers, the root cause is instantly obvious. TSAN is more problematic to run in production; it's a _lot_ slower.
- boris 8y agoYes, thread-sanitized code is unbearably slow. Plus, you cannot use both ASAN and TSAN in the same build (but either can be combined with UBSAN).
- muricula 8y agoYesterday a coworker had an issue where ASAN wasn't finding a std::vector out of bounds error. It turns out the code doing the deference was in libc++, which was not ASAN instrumented even though his program was. Valgrind found the error right away. They're all great tools with their own pros and cons. If you have the testing infrastructure, use all of them.