8 ms·

Does anyone use any of these tools as part of their professional life? I would love to know which tool you use and how it's used.

by fusslo 3y ago

Alifatisk 3y ago
Someone mentioned Infer (by Meta) some years ago, now when checking it out, it looks to be dead. Otherwise, I've used flawfinder for some of my projects.
- retrocryptid 3y ago
  When I write my autobiography, an entire chapter will be named "/* FlawFinder: ignore */".
  - sumtechguy 3y ago
    I used to use a program that some devs were convinced had tons of false positives. Until you actually look at it. Usually something silly was going on to either cause it to not see what you were doing (rare) or there really was an issue (not rare). The 'oh that is a false positive' usually clues me into there is a whole class of bugs that this project has that need some deep looking into. Now I am not saying that happened to you but sometimes they are worth looking at. I saw one yesterday the dev did not understand why it would randomly crash. His destruction loop was backwards and was using pointers that may or may not still be good. Sometimes you cannot control the lib. So you have to 'ignore' but that only gives you time to mitigate.
flohofwoe 3y ago
Not professionaly, because currently I mostly write Typescript in my job, but I run all my (hobby) C and C++ code through clang static analyzer (exceptionally simple in Xcode because it's fully integrated, but also in command line builds - I haven't bothered with setting up a CI for this yet though, still on the todo list). For command line builds I'm planning to migrate to clang-tidy though (contrary to popular belief clang-tidy is not just a linter, it also runs the clang static analyzer checks). Visual Studio also comes with their own static analyzer, but this doesn't seem to be as thorough as Clang's. Also: static analyzers are just one aspect, the other is runtime sanitizers (e.g. clang's ASAN, UBSAN and TSAN).
aeonik 3y ago
I'm pretty sure everyone uses a basic static analyzer for code completion, formatting, and syntax. Most IDEs just set it up for you. I've used the "professional" security ones including TrustInSoft and Synopsis tools. They are pretty cool, but give a lot of false positives, you either need teams to really understand the security model of these systems, or to have a dedicated team manage and triage things. They do a great job catching some egregious mistakes though. That being said, I think Rust is takes the better approach of trying to build safety directly into the language and compiler. Though I think there is still a lot of room for innovation here.
- Alifatisk 3y ago
  > I'm pretty sure everyone uses a basic static analyzer for code completion, formatting, and syntax. Most IDEs just set it up for you. Yeah, this extension is almost the default choice for vscode https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools https://marketplace.visualstudio.com/items?itemName=ms-vscod...
  - flohofwoe 3y ago
    As far as I'm aware the vanilla Intellisense (e.g. error squiggles and warnings while typing) isn't doing proper static analysis only simple syntax error checks. And while clang-tidy is integrated with the MS C++ extension, it needs to be run manually, or automatically on file save/open (see: https://devblogs.microsoft.com/cppblog/visual-studio-code-c-december-2021-update-clang-tidy/ https://devblogs.microsoft.com/cppblog/visual-studio-code-c-...) - I guess it's too expensive to just run in the background while typing.
    - pjmlp 3y ago
      Nowadays that isn't any longer the case, linting is also performed, at least on Visual Studio proper. https://learn.microsoft.com/en-us/cpp/ide/cpp-linter-overview?view=msvc-170 https://learn.microsoft.com/en-us/cpp/ide/cpp-linter-overvie...
- flohofwoe 3y ago
deleted 3y ago
[deleted]
pjmlp 3y ago
SonarQube and Visual Studio analyser for the most part, also enabling bounds checking and other test configurations. In the old days I used stuff like Insure++. Additionally when using C is unavoidable, taking a abstract data types design approach[0], to minimize killing data structure invariants as much as possible. [0] - https://en.wikipedia.org/wiki/Abstract_data_type https://en.wikipedia.org/wiki/Abstract_data_type
rwmj 3y ago
Looking at the C section, clang-tidy, gcc analyzer, -Werror, and Coverity. I use several more non-static (runtime) checkers too. I tried to use Frama-C a while back and gave a talk about it, but for us it wasn't too practical: http://oirase.annexia.org/tmp/2020-frama-c-tech-talk.mp4 http://oirase.annexia.org/tmp/2020-frama-c-tech-talk.mp4
AlotOfReading 3y ago
I write safety critical code, so I've worked with many of these: frama-C, TIS, Polyspace, KLEE, CBMC, and some that aren't on the list like RV-match, axivion, and sonarqube. With virtually any tool in this category, you need to spend a good amount of time ensuring it's integrated properly at the center of your workflows. They all have serious caveats and usually false positives that need to be addressed before you develop warning blindness. You also need to be extremely careful about what guarantees they're actually giving you and read any associated formal methods documentation very carefully. Even a tool that formally proves the absence of certain classes of errors only does so under certain preconditions, which may not encompass everything you're supporting. I once found a theoretical issue in a project that had a nice formal proof because the semantics of some of the tooling implicitly assumed C99, so certain conforming (but stupid) C89 implementations could violate the guarantees. But some notes: Frama-C is a nice design, but installing it is a pain, ACSL is badly documented/hard to learn, and you really want the proprietary stuff for anything nontrivial. TrustInSoft (TIS) has nice enterprise tooling, but their "open source" stuff isn't worth bothering with. Also, they make it impossible to introduce to a company without going through the enterprise sales funnel, so I've never successfully gotten it introduced because they suck at enterprise sales compared to e.g. Mathworks. RV-match (https://runtimeverification.com/match https://runtimeverification.com/match) is what I've adopted for my open source code that needs tooling beyond sanitizers because it's nice to work with and has reasonable detection rates without too many false positives. Polyspace, sonarqube, axivion, compiler warnings, etc are all pretty subpar tools in my experience. I've never seen them work well in this space. They're usually either not detecting anything useful or falsely erroring on obviously harmless constructs, often both.
- KsassPeuk 3y ago
  > ACSL is badly documented/hard to learn, and you really want the proprietary stuff for anything nontrivial. That's interesting. What kind of documentation would you expect in addition to ACSL by Example and the WP tutorial? What are the proprietary components that you miss the most?
- ChrisRR 3y ago
  > but installing it is a pain See every embedded dev tool
amadvance 3y ago
The Clang static analyzer is integrated into the build pipeline. Any warnings will cause the build to fail. Additionally, build with the flags -Wall and -Werror. When testing, run with runtime checkers such as Valgrind and sanitizers. Periodically, run other static analyzers like Klocwork and Coverity. They can catch many more issues than Clang. It's not that Clang is bad, but it has inherent limitations because it only analyzes a single source file and stops analysis when you call a function from another module
- Joel_Mckay 3y ago
  Most LLVM projects I see are simply replacing -O3 pipelines. i.e. the code has already been heavily stress-tested prior to a safer optimization port of a stripped binary. I also do time critical stuff, so llvm is a nonstarter for predictive latency in code motion. For most other use-cases, clang/llvm typically does improve performance a bit, and I do like its behavior on ARM. Happy coding =)
- yaantc 3y ago
  > It's not that Clang is bad, but it has inherent limitations because it only analyzes a single source file and stops analysis when you call a function from another module. Nowadays that's only the default. But you can enable "cross translation units" [1] support to perform analysis across all the files of an application. It's easier to deploy CTU by using CodeChecker [2]. Also for the Clang static analyzer: make sure the build does use Z3. It should be the case now in most distro (it's the case in Debian stable ;). It will improve the results. With both CTU and Z3 I'm very happy with the results. Klocwork mostly only reported false alarms after a clean CodeChecker pass. [1] https://clang.llvm.org/docs/analyzer/user-docs/CrossTranslationUnit.html [2] https://codechecker.readthedocs.io/en/latest/
  - HybridCurve 3y ago
    I agree that CTU analysis makes it better. There are also a bunch of tunables for the clang analyzer that you can take advantage of that suites like CodeChecker do not fully allow access to or take advantage of.
bluGill 3y ago
I use a number of tools. The most important consideration is a zero false positives policy. If there are any flase positives people soon learn to ignore all errors. I've traced production bugs to code where immediately above the line at fault was a comment to turn off the static checker for the next line. Someone didn't check to see if the tool was right because experience is the tool was normally wrong. Don't forget that real errors you fix will disappear so even a few false positives are bad because you see them all the time and so they seem worse than they are. Now that our tools have a zero false positive policy we can require all attempts to silence warnings are allowed only after a bug report is opened with the tool. Everytime we upgrade we grep and remove the silence warning comments for things fixed.
- ChrisRR 3y ago
  Most of my lint warnings aren't false positives though, they're potential issues. Most of my time sorting through lint warnings is addressing the warnings for potential issues. The potential for an issue may be legitimate, but that doesn't make it a false positive.
  - bluGill 3y ago
    Some lint tools are better than others about it. Potential is not enough - there needs to be a clear fix that shuts up the warning and makes the code function as intended (which may be different than as I wrote it!). The tool I'm thinking of didn't have that, so we regularly silenced warnings, and then a few years of that and the habit became to not even bother looking at what the warning was, just silence it. If there is a clear fix that makes the code objectively better then I'm for the tool. However if it warns about correct code because of a potential issue that doesn't even apply that is a false positive.
Joel_Mckay 3y ago
Not often, I prefer long-term leak-testing under load, as most libraries and gui frameworks still slowly leak memory in time (many reasons, and not C/C++ specific). Simply using g++ to compile C can clean up the formatting a bit, and Valgrind can be helpful locating shit code. One should use a fault tolerant design pattern... especially in C. i.e. assume stuff will break, interleave guard variables, and use buffer safe versions of standard string functions. There are also several back-ported data-structure libraries for C that make it less goofy to handle. Personally I like designing systems like an insect colony, avoid thread safety issues, and constantly life-cycle individual modules. This makes things easy to scale/audit, kinder to peers who read 2 pages to understand the codes purpose, and inherently robust as locality is unnecessary to continue to function (parts can fail back and self-repair while running). Have fun. =)
yaantc 3y ago
Clang static analyzer and tidy through CodeChecker [1]. Plus the compiler warnings of course. This to complement unit and integration tests in simulation, with all sanitizers (UBSAN, ASAN, MSAN, TSAN). Both are useful. In practice, the runtime tests catch most bugs. But now and then some bug sleeps through and is caught by the Clang static analyzer. It's always impressive to see a display of 30+ steps leading from an initial condition to a bug. Finding bugs is the the static analyzer really. Clang tidy (a linter) is for code "cleanliness" and avoiding dangerous constructs. But I don't remember tidy finding a real issue, contrary to the static analyzer. [1] https://codechecker.readthedocs.io/en/latest/ https://codechecker.readthedocs.io/en/latest/

Does anyone use any of these tools as part of their professional life? I would love to know which tool you use and how it's used.

by fusslo 3y ago

Does anyone use any of these tools as part of their professional life? I would love to know which tool you use and how it's used.

Alifatisk 3y ago

Someone mentioned Infer (by Meta) some years ago, now when checking it out, it looks to be dead. Otherwise, I've used flawfinder for some of my projects.

retrocryptid 3y ago
When I write my autobiography, an entire chapter will be named "/* FlawFinder: ignore */".
- sumtechguy 3y ago
  I used to use a program that some devs were convinced had tons of false positives. Until you actually look at it. Usually something silly was going on to either cause it to not see what you were doing (rare) or there really was an issue (not rare). The 'oh that is a false positive' usually clues me into there is a whole class of bugs that this project has that need some deep looking into. Now I am not saying that happened to you but sometimes they are worth looking at. I saw one yesterday the dev did not understand why it would randomly crash. His destruction loop was backwards and was using pointers that may or may not still be good. Sometimes you cannot control the lib. So you have to 'ignore' but that only gives you time to mitigate.

flohofwoe 3y ago

Not professionaly, because currently I mostly write Typescript in my job, but I run all my (hobby) C and C++ code through clang static analyzer (exceptionally simple in Xcode because it's fully integrated, but also in command line builds - I haven't bothered with setting up a CI for this yet though, still on the todo list). For command line builds I'm planning to migrate to clang-tidy though (contrary to popular belief clang-tidy is not just a linter, it also runs the clang static analyzer checks). Visual Studio also comes with their own static analyzer, but this doesn't seem to be as thorough as Clang's. Also: static analyzers are just one aspect, the other is runtime sanitizers (e.g. clang's ASAN, UBSAN and TSAN).

aeonik 3y ago

I'm pretty sure everyone uses a basic static analyzer for code completion, formatting, and syntax. Most IDEs just set it up for you. I've used the "professional" security ones including TrustInSoft and Synopsis tools. They are pretty cool, but give a lot of false positives, you either need teams to really understand the security model of these systems, or to have a dedicated team manage and triage things. They do a great job catching some egregious mistakes though. That being said, I think Rust is takes the better approach of trying to build safety directly into the language and compiler. Though I think there is still a lot of room for innovation here.

Alifatisk 3y ago
> I'm pretty sure everyone uses a basic static analyzer for code completion, formatting, and syntax. Most IDEs just set it up for you. Yeah, this extension is almost the default choice for vscode https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools https://marketplace.visualstudio.com/items?itemName=ms-vscod...
- flohofwoe 3y ago
  As far as I'm aware the vanilla Intellisense (e.g. error squiggles and warnings while typing) isn't doing proper static analysis only simple syntax error checks. And while clang-tidy is integrated with the MS C++ extension, it needs to be run manually, or automatically on file save/open (see: https://devblogs.microsoft.com/cppblog/visual-studio-code-c-december-2021-update-clang-tidy/ https://devblogs.microsoft.com/cppblog/visual-studio-code-c-...) - I guess it's too expensive to just run in the background while typing.
  - pjmlp 3y ago
    Nowadays that isn't any longer the case, linting is also performed, at least on Visual Studio proper. https://learn.microsoft.com/en-us/cpp/ide/cpp-linter-overview?view=msvc-170 https://learn.microsoft.com/en-us/cpp/ide/cpp-linter-overvie...
flohofwoe 3y ago

deleted 3y ago

[deleted]

pjmlp 3y ago

SonarQube and Visual Studio analyser for the most part, also enabling bounds checking and other test configurations. In the old days I used stuff like Insure++. Additionally when using C is unavoidable, taking a abstract data types design approach[0], to minimize killing data structure invariants as much as possible. [0] - https://en.wikipedia.org/wiki/Abstract_data_type https://en.wikipedia.org/wiki/Abstract_data_type

rwmj 3y ago

Looking at the C section, clang-tidy, gcc analyzer, -Werror, and Coverity. I use several more non-static (runtime) checkers too. I tried to use Frama-C a while back and gave a talk about it, but for us it wasn't too practical: http://oirase.annexia.org/tmp/2020-frama-c-tech-talk.mp4 http://oirase.annexia.org/tmp/2020-frama-c-tech-talk.mp4

AlotOfReading 3y ago

I write safety critical code, so I've worked with many of these: frama-C, TIS, Polyspace, KLEE, CBMC, and some that aren't on the list like RV-match, axivion, and sonarqube. With virtually any tool in this category, you need to spend a good amount of time ensuring it's integrated properly at the center of your workflows. They all have serious caveats and usually false positives that need to be addressed before you develop warning blindness. You also need to be extremely careful about what guarantees they're actually giving you and read any associated formal methods documentation very carefully. Even a tool that formally proves the absence of certain classes of errors only does so under certain preconditions, which may not encompass everything you're supporting. I once found a theoretical issue in a project that had a nice formal proof because the semantics of some of the tooling implicitly assumed C99, so certain conforming (but stupid) C89 implementations could violate the guarantees. But some notes: Frama-C is a nice design, but installing it is a pain, ACSL is badly documented/hard to learn, and you really want the proprietary stuff for anything nontrivial. TrustInSoft (TIS) has nice enterprise tooling, but their "open source" stuff isn't worth bothering with. Also, they make it impossible to introduce to a company without going through the enterprise sales funnel, so I've never successfully gotten it introduced because they suck at enterprise sales compared to e.g. Mathworks. RV-match (https://runtimeverification.com/match https://runtimeverification.com/match) is what I've adopted for my open source code that needs tooling beyond sanitizers because it's nice to work with and has reasonable detection rates without too many false positives. Polyspace, sonarqube, axivion, compiler warnings, etc are all pretty subpar tools in my experience. I've never seen them work well in this space. They're usually either not detecting anything useful or falsely erroring on obviously harmless constructs, often both.

KsassPeuk 3y ago
> ACSL is badly documented/hard to learn, and you really want the proprietary stuff for anything nontrivial. That's interesting. What kind of documentation would you expect in addition to ACSL by Example and the WP tutorial? What are the proprietary components that you miss the most?
ChrisRR 3y ago
> but installing it is a pain See every embedded dev tool

amadvance 3y ago

The Clang static analyzer is integrated into the build pipeline. Any warnings will cause the build to fail. Additionally, build with the flags -Wall and -Werror. When testing, run with runtime checkers such as Valgrind and sanitizers. Periodically, run other static analyzers like Klocwork and Coverity. They can catch many more issues than Clang. It's not that Clang is bad, but it has inherent limitations because it only analyzes a single source file and stops analysis when you call a function from another module

Joel_Mckay 3y ago
Most LLVM projects I see are simply replacing -O3 pipelines. i.e. the code has already been heavily stress-tested prior to a safer optimization port of a stripped binary. I also do time critical stuff, so llvm is a nonstarter for predictive latency in code motion. For most other use-cases, clang/llvm typically does improve performance a bit, and I do like its behavior on ARM. Happy coding =)
yaantc 3y ago
> It's not that Clang is bad, but it has inherent limitations because it only analyzes a single source file and stops analysis when you call a function from another module. Nowadays that's only the default. But you can enable "cross translation units" [1] support to perform analysis across all the files of an application. It's easier to deploy CTU by using CodeChecker [2]. Also for the Clang static analyzer: make sure the build does use Z3. It should be the case now in most distro (it's the case in Debian stable ;). It will improve the results. With both CTU and Z3 I'm very happy with the results. Klocwork mostly only reported false alarms after a clean CodeChecker pass. [1] https://clang.llvm.org/docs/analyzer/user-docs/CrossTranslationUnit.html [2] https://codechecker.readthedocs.io/en/latest/
- HybridCurve 3y ago
  I agree that CTU analysis makes it better. There are also a bunch of tunables for the clang analyzer that you can take advantage of that suites like CodeChecker do not fully allow access to or take advantage of.

bluGill 3y ago

I use a number of tools. The most important consideration is a zero false positives policy. If there are any flase positives people soon learn to ignore all errors. I've traced production bugs to code where immediately above the line at fault was a comment to turn off the static checker for the next line. Someone didn't check to see if the tool was right because experience is the tool was normally wrong. Don't forget that real errors you fix will disappear so even a few false positives are bad because you see them all the time and so they seem worse than they are. Now that our tools have a zero false positive policy we can require all attempts to silence warnings are allowed only after a bug report is opened with the tool. Everytime we upgrade we grep and remove the silence warning comments for things fixed.

ChrisRR 3y ago
Most of my lint warnings aren't false positives though, they're potential issues. Most of my time sorting through lint warnings is addressing the warnings for potential issues. The potential for an issue may be legitimate, but that doesn't make it a false positive.
- bluGill 3y ago
  Some lint tools are better than others about it. Potential is not enough - there needs to be a clear fix that shuts up the warning and makes the code function as intended (which may be different than as I wrote it!). The tool I'm thinking of didn't have that, so we regularly silenced warnings, and then a few years of that and the habit became to not even bother looking at what the warning was, just silence it. If there is a clear fix that makes the code objectively better then I'm for the tool. However if it warns about correct code because of a potential issue that doesn't even apply that is a false positive.

Joel_Mckay 3y ago

Not often, I prefer long-term leak-testing under load, as most libraries and gui frameworks still slowly leak memory in time (many reasons, and not C/C++ specific). Simply using g++ to compile C can clean up the formatting a bit, and Valgrind can be helpful locating shit code. One should use a fault tolerant design pattern... especially in C. i.e. assume stuff will break, interleave guard variables, and use buffer safe versions of standard string functions. There are also several back-ported data-structure libraries for C that make it less goofy to handle. Personally I like designing systems like an insect colony, avoid thread safety issues, and constantly life-cycle individual modules. This makes things easy to scale/audit, kinder to peers who read 2 pages to understand the codes purpose, and inherently robust as locality is unnecessary to continue to function (parts can fail back and self-repair while running). Have fun. =)

yaantc 3y ago

Clang static analyzer and tidy through CodeChecker [1]. Plus the compiler warnings of course. This to complement unit and integration tests in simulation, with all sanitizers (UBSAN, ASAN, MSAN, TSAN). Both are useful. In practice, the runtime tests catch most bugs. But now and then some bug sleeps through and is caught by the Clang static analyzer. It's always impressive to see a display of 30+ steps leading from an initial condition to a bug. Finding bugs is the the static analyzer really. Clang tidy (a linter) is for code "cleanliness" and avoiding dangerous constructs. But I don't remember tidy finding a real issue, contrary to the static analyzer. [1] https://codechecker.readthedocs.io/en/latest/ https://codechecker.readthedocs.io/en/latest/