6 ms·

I get that this is a list of static analyzers, but for C runtime checkers are useful to know about too. Valgrind, clang ASAN, american-fuzzy-lop, glibc _FORTIF

by rwmj 3y ago

I get that this is a list of static analyzers, but for C runtime checkers are useful to know about too. Valgrind, clang ASAN, american-fuzzy-lop, glibc _FORTIFY_SOURCE are all useful tools in the toolkit.

Alifatisk 3y ago
I might actually use Valgrind in the future! Looks helpful as hell. https://valgrind.org/docs/manual/quick-start.html https://valgrind.org/docs/manual/quick-start.html
- rwmj 3y ago
  valgrind is really essential. Just about every C project that I have written has a "make check-valgrind" rule which runs the same set of tests as "make check", but under valgrind. Passing both is a requirement for release (although for various reasons to do with cost, only "make check" is done in CI). valgrind mainly finds memory leaks, buffer overflows and use of unintialized memory. It even works relatively well with some non-C languages. Example: https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0acb2ef8887709c47d3cc/tests/Makefile.am#L159-L161 https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0...
  - gpderetta 3y ago
    Valgrind was quite essential, but these days the various sanitizers are both easier to integrate, faster and catch more issues.
    - adrianN 3y ago
      Do they catch more issues? I thought valgrind was the gold standard. I would be happy to be proven wrong!
      - PhilipRoman 3y ago
        Its a tradeoff, they both specialize for slightly different situations. Sanitizers have intricate knowledge of what is undefined behavior at a source level. For example signed integer overflow, etc. Some of these are simply not possible to determine at runtime, as a valid C program is allowed to generate the same machine code as one containing UB. Valgrind, on the other hand, cares purely about the machine code (with a few exceptions like malloc, free, etc.). It can peek into your libraries even if they aren't compiled with sanitizer flags. It is especially good for finding uninitialized memory usage - IIRC it can track this data even on the level of individual bits in a register.
- hannob 3y ago
  A combination of testing with ASAN and MSAN is pretty much always better than valgrind if you have source code. Though MSAN use can be somewhat challenging, as you need to have all dependencies built with MSAN, too.
  - rwmj 3y ago
    It's different. The reason valgrind might be better is that it tests production binaries. Having to recompile everything with ASAN is difficult in traditional 'make && make check' workflows, or if you want to build and distribute tested binaries (although can work in CI if set up in advance, and we do use it sometimes there). ASAN + fuzzing is a great combination.
- williamcotton 3y ago
  I would recommend getting familiar with making a suppressions file! Valgrind is not perfect and it can be a little noisy with “possible” leaks, depending on what you’re doing!
- jbaber 3y ago
  As others have said, it's indisposable. It's useful to learn how to suppress false positives (e.g. from boost) so you can see when an actual leak appears.
gavinhoward 3y ago
Yes, absolutely. Static analyzers are for when your test suite is subpar. If you're obsessive, like the SQLite guys [1], runtime checkers are better. But you have to be obsessive about an excessive test suite. [1]: https://sqlite.org/faq.html#q17 https://sqlite.org/faq.html#q17
- AlotOfReading 3y ago
  Yeah, no. They're different things. Formal methods, including static analyzers, can prove the absence of issues in all possible conditions under some assumptions. Tests verify that implementation, under the tested conditions, once. You can't replace testing with formal methods, but formal methods are more powerful when they work. That sqlite hasn't found them useful is just a reflection of how absurdly far out they are on the testing spectrum and how complicated the problem domain they're operating in is (very).
  - gavinhoward 3y ago
    Don't get me wrong; I'm a fan of formal methods, when they can work. But you're also weakening your own point; you admit that if you test enough, they don't work as well, and you also admit that they work less well the more complicated things get. The other problem is showing that the artifact you proved is the same artifact that you are using. The enormous amount of effort put in by the seL4 people is proof that that is non-trivial, and seL4 is tiny compared to SQLite. So when you say, "formal methods are more powerful when they work," you are right, but "when they work" is carrying a lot of weight.
    - nicoburns 3y ago
      Stronger type systems can get you a lot of the benefits of formal methods with very little work. A SQLite level test suite is probably still better, but very few projects have the time or budget to do that.
      - gavinhoward 3y ago
ChrisRR 3y ago
Given how much C is used in embedded environments, runtime checking is often not an option
- mumblemumble 3y ago
  I would guess that even when you're targeting embedded environments you can still use runtime checks to help find problems during any part of the development lifecycle where the code isn't running on the target hardware? That's roughly what I do for certain non-embedded work with high performance needs. There's a lot of contract checking and assertions that are too heavyweight for production, so they aren't included in deployment builds. But I certainly wouldn't want to not test at all just because I can't test in production.
  - ChrisRR 3y ago
    You can, but there's not many tools that you can use to just throw everything at the code like with desktop software. Even then it's often very dependant on the MCU and compilers you're using
  - Gibbon1 3y ago
    I try to do runtime checks for my embedded code. Because as I like to say there isn't anyone to press F5 when something barfs. Bigger issue for me is what to do when a run time check fails. Sometimes there is nothing to be done but to log the error and reset. Other times, you log the error, clean up and keep going. Which why I dislike rust's run time checks which panic without letting the program try to recover.
mumblemumble 3y ago
They keep the dynamic analysis tools in a separate repository: https://github.com/analysis-tools-dev/dynamic-analysis https://github.com/analysis-tools-dev/dynamic-analysis Both repos link each other close to the tops of their respective readmes. Annoyingly, though, the webstite seems to only include the static tools.
- mre 3y ago
  Maintainer here. Thought we fixed that. I'll look into it.

by rwmj 3y ago

Alifatisk 3y ago

I might actually use Valgrind in the future! Looks helpful as hell. https://valgrind.org/docs/manual/quick-start.html https://valgrind.org/docs/manual/quick-start.html

rwmj 3y ago
valgrind is really essential. Just about every C project that I have written has a "make check-valgrind" rule which runs the same set of tests as "make check", but under valgrind. Passing both is a requirement for release (although for various reasons to do with cost, only "make check" is done in CI). valgrind mainly finds memory leaks, buffer overflows and use of unintialized memory. It even works relatively well with some non-C languages. Example: https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0acb2ef8887709c47d3cc/tests/Makefile.am#L159-L161 https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0...
- gpderetta 3y ago
  Valgrind was quite essential, but these days the various sanitizers are both easier to integrate, faster and catch more issues.
  - adrianN 3y ago
    Do they catch more issues? I thought valgrind was the gold standard. I would be happy to be proven wrong!
    - PhilipRoman 3y ago
      Its a tradeoff, they both specialize for slightly different situations. Sanitizers have intricate knowledge of what is undefined behavior at a source level. For example signed integer overflow, etc. Some of these are simply not possible to determine at runtime, as a valid C program is allowed to generate the same machine code as one containing UB. Valgrind, on the other hand, cares purely about the machine code (with a few exceptions like malloc, free, etc.). It can peek into your libraries even if they aren't compiled with sanitizer flags. It is especially good for finding uninitialized memory usage - IIRC it can track this data even on the level of individual bits in a register.
hannob 3y ago
A combination of testing with ASAN and MSAN is pretty much always better than valgrind if you have source code. Though MSAN use can be somewhat challenging, as you need to have all dependencies built with MSAN, too.
- rwmj 3y ago
  It's different. The reason valgrind might be better is that it tests production binaries. Having to recompile everything with ASAN is difficult in traditional 'make && make check' workflows, or if you want to build and distribute tested binaries (although can work in CI if set up in advance, and we do use it sometimes there). ASAN + fuzzing is a great combination.
williamcotton 3y ago
I would recommend getting familiar with making a suppressions file! Valgrind is not perfect and it can be a little noisy with “possible” leaks, depending on what you’re doing!
jbaber 3y ago
As others have said, it's indisposable. It's useful to learn how to suppress false positives (e.g. from boost) so you can see when an actual leak appears.

gavinhoward 3y ago

Yes, absolutely. Static analyzers are for when your test suite is subpar. If you're obsessive, like the SQLite guys [1], runtime checkers are better. But you have to be obsessive about an excessive test suite. [1]: https://sqlite.org/faq.html#q17 https://sqlite.org/faq.html#q17

AlotOfReading 3y ago
Yeah, no. They're different things. Formal methods, including static analyzers, can prove the absence of issues in all possible conditions under some assumptions. Tests verify that implementation, under the tested conditions, once. You can't replace testing with formal methods, but formal methods are more powerful when they work. That sqlite hasn't found them useful is just a reflection of how absurdly far out they are on the testing spectrum and how complicated the problem domain they're operating in is (very).
- gavinhoward 3y ago
  Don't get me wrong; I'm a fan of formal methods, when they can work. But you're also weakening your own point; you admit that if you test enough, they don't work as well, and you also admit that they work less well the more complicated things get. The other problem is showing that the artifact you proved is the same artifact that you are using. The enormous amount of effort put in by the seL4 people is proof that that is non-trivial, and seL4 is tiny compared to SQLite. So when you say, "formal methods are more powerful when they work," you are right, but "when they work" is carrying a lot of weight.
  - nicoburns 3y ago
    Stronger type systems can get you a lot of the benefits of formal methods with very little work. A SQLite level test suite is probably still better, but very few projects have the time or budget to do that.
    - gavinhoward 3y ago

ChrisRR 3y ago

Given how much C is used in embedded environments, runtime checking is often not an option

mumblemumble 3y ago
I would guess that even when you're targeting embedded environments you can still use runtime checks to help find problems during any part of the development lifecycle where the code isn't running on the target hardware? That's roughly what I do for certain non-embedded work with high performance needs. There's a lot of contract checking and assertions that are too heavyweight for production, so they aren't included in deployment builds. But I certainly wouldn't want to not test at all just because I can't test in production.
- ChrisRR 3y ago
  You can, but there's not many tools that you can use to just throw everything at the code like with desktop software. Even then it's often very dependant on the MCU and compilers you're using
- Gibbon1 3y ago
  I try to do runtime checks for my embedded code. Because as I like to say there isn't anyone to press F5 when something barfs. Bigger issue for me is what to do when a run time check fails. Sometimes there is nothing to be done but to log the error and reset. Other times, you log the error, clean up and keep going. Which why I dislike rust's run time checks which panic without letting the program try to recover.

mumblemumble 3y ago

They keep the dynamic analysis tools in a separate repository: https://github.com/analysis-tools-dev/dynamic-analysis https://github.com/analysis-tools-dev/dynamic-analysis Both repos link each other close to the tops of their respective readmes. Annoyingly, though, the webstite seems to only include the static tools.

mre 3y ago
Maintainer here. Thought we fixed that. I'll look into it.