4 ms·

Selling static analyzers are hard. Your customer either has a bunch of code, none of which was written with the static analyzer in mind, and so they have to spe

Selling static analyzers are hard. Your customer either has a bunch of code, none of which was written with the static analyzer in mind, and so they have to spend a bunch of time fixing spurious errors and warnings, or they have no code, in which case they probably also have no money to pay you with.

NordSteve 5y ago
I've implemented static analyzers in several codebases, and every time it's paid off in finding multiple "must fix" bugs.
acdha 5y ago
> so they have to spend a bunch of time fixing spurious errors and warnings It's rare for a tool to be so shoddy that this is all you get. From what I've seen, you'll get three categories of report: valid bugs, valid bugs which the developers insist are false positives until forced to do in-depth analysis, and actual false-positives. The exact ratios vary depending on the tool in question but generally I've seen the first two categories be a good bit larger than the third.
- colinmhayes 5y ago
  My last job was a fortune 500 that decided to start using a static analyzer on a 20 year old project. Thousands of false positives and less than ten true positives after hundreds of hours were spent going through it. Now the codebase was definitely a piece of work which is why there were so many false positives, but there were plenty of "the static analyzer is horrible" false positives too.
  - acdha 5y ago
    Yes - I’m not saying that you won’t see false-positives, but I’ve rarely seen a project where it didn’t find real issues, including some that people resisted accepting (including exploitable security holes). If you approach it determined to reject everything, or to blindly follow the tool, you’ll get poor outcomes. If you view it as a tool which can help you improve your code and triage results accordingly, you’ll almost always have better results.
  - naasking 5y ago
    > Thousands of false positives and less than ten true positives after hundreds of hours were spent going through it. Ten true positives could very well be worth thousands of programmer hours if they're severe enough.
- geofft 5y ago
  I think this depends on two things: 1) Quality of existing language/compiler. Some languages are more prone to turning carelessness into exploitable bugs than others. Some compilers are better at flagging questionable practices than others (and, as an approximation, newer compilers for basically any compiled language are better than older ones). 2) How much you care about fixing the problems. If a team is understaffed and asked to overdeliver, they have no time for fixing technical debt of any sort, whether static analysis warnings or old cryptographic hashes or whatever. Static analysis tools do best when 1 is low and 2 is high. But that's unusual. First, you never start a project in that situation. If you care about the sort of bugs a static analyzer will find, you almost certainly are going to pick a good ecosystem to implement in. So you're working with legacy code, almost by definition. And if you have the bandwidth to address tech debt, there are many ways to use better toolchains and deal with legacy code. And this isn't a "Rust solves everything" post - frankly, upgrading your C/C++ compiler and listening to what it says will get you a lot of the way there. But that itself tends to be a sizable tech-debt-reduction project. (I'm one of the people on the GCC upgrade team at my own workplace, and we do justify the work to senior management by pointing out that new warnings are helpful for code quality, and we do spend time addressing those warnings.) We also have more tools and frameworks for moving parts of your computation to a different language or toolchain (more norms around microservices, better serialization libraries, etc.) if you want to go the rewrite route but want to do it incrementally. And there are plenty of language choices for a rewrite; frankly, most use cases in the '90s that needed C++ will do just fine today with unoptimized and easy-to-read Python. So, if 2 is high, you have a lot of options other than static analysis. (Aside: the post is about PL startups. I would bet the average PL grad would rather work with the ecosystem of a new, modern programming languages that has benefited from recent research than write tools to work around an old and known-suboptimal one.) I think the only real case where static analysis can work is where 1 is low but the choice to buy the static analysis tool causes 2 to become high. Basically, it acts like a consultant. The devs already know, vaguely, that the code is crap, and they wish someone would give them resources to fix it. Senior management buys a shiny static analysis tool, which is an objective outside voice saying how bad the code is. They tell devs to fix it, and devs are happy to knock out obvious fixes, your category 1 - and they'll still debate the non-obvious fixes, your category 2, because they're not being motivated by the static analysis tool, they're motivated by their own sense of what code is crappy, they just know that their new OKR is to silence all the static analysis warnings. That's certainly a better outcome than not fixing anything, but that's still a worse outcome for the company, probably, than asking your devs what's wrong with the code and how they want to fix it.
fredophile 5y ago
I'd day there's another big group you're missing. People with large codebases that compile with tons of warnings that just get ignored. If you aren't willing to spend the time fixing the existing warnings to prevent problems why would you spend money on a tool that will give you more of them?

NordSteve 5y ago

I've implemented static analyzers in several codebases, and every time it's paid off in finding multiple "must fix" bugs.

acdha 5y ago

> so they have to spend a bunch of time fixing spurious errors and warnings It's rare for a tool to be so shoddy that this is all you get. From what I've seen, you'll get three categories of report: valid bugs, valid bugs which the developers insist are false positives until forced to do in-depth analysis, and actual false-positives. The exact ratios vary depending on the tool in question but generally I've seen the first two categories be a good bit larger than the third.

colinmhayes 5y ago
My last job was a fortune 500 that decided to start using a static analyzer on a 20 year old project. Thousands of false positives and less than ten true positives after hundreds of hours were spent going through it. Now the codebase was definitely a piece of work which is why there were so many false positives, but there were plenty of "the static analyzer is horrible" false positives too.
- acdha 5y ago
  Yes - I’m not saying that you won’t see false-positives, but I’ve rarely seen a project where it didn’t find real issues, including some that people resisted accepting (including exploitable security holes). If you approach it determined to reject everything, or to blindly follow the tool, you’ll get poor outcomes. If you view it as a tool which can help you improve your code and triage results accordingly, you’ll almost always have better results.
- naasking 5y ago
  > Thousands of false positives and less than ten true positives after hundreds of hours were spent going through it. Ten true positives could very well be worth thousands of programmer hours if they're severe enough.
geofft 5y ago
I think this depends on two things: 1) Quality of existing language/compiler. Some languages are more prone to turning carelessness into exploitable bugs than others. Some compilers are better at flagging questionable practices than others (and, as an approximation, newer compilers for basically any compiled language are better than older ones). 2) How much you care about fixing the problems. If a team is understaffed and asked to overdeliver, they have no time for fixing technical debt of any sort, whether static analysis warnings or old cryptographic hashes or whatever. Static analysis tools do best when 1 is low and 2 is high. But that's unusual. First, you never start a project in that situation. If you care about the sort of bugs a static analyzer will find, you almost certainly are going to pick a good ecosystem to implement in. So you're working with legacy code, almost by definition. And if you have the bandwidth to address tech debt, there are many ways to use better toolchains and deal with legacy code. And this isn't a "Rust solves everything" post - frankly, upgrading your C/C++ compiler and listening to what it says will get you a lot of the way there. But that itself tends to be a sizable tech-debt-reduction project. (I'm one of the people on the GCC upgrade team at my own workplace, and we do justify the work to senior management by pointing out that new warnings are helpful for code quality, and we do spend time addressing those warnings.) We also have more tools and frameworks for moving parts of your computation to a different language or toolchain (more norms around microservices, better serialization libraries, etc.) if you want to go the rewrite route but want to do it incrementally. And there are plenty of language choices for a rewrite; frankly, most use cases in the '90s that needed C++ will do just fine today with unoptimized and easy-to-read Python. So, if 2 is high, you have a lot of options other than static analysis. (Aside: the post is about PL startups. I would bet the average PL grad would rather work with the ecosystem of a new, modern programming languages that has benefited from recent research than write tools to work around an old and known-suboptimal one.) I think the only real case where static analysis can work is where 1 is low but the choice to buy the static analysis tool causes 2 to become high. Basically, it acts like a consultant. The devs already know, vaguely, that the code is crap, and they wish someone would give them resources to fix it. Senior management buys a shiny static analysis tool, which is an objective outside voice saying how bad the code is. They tell devs to fix it, and devs are happy to knock out obvious fixes, your category 1 - and they'll still debate the non-obvious fixes, your category 2, because they're not being motivated by the static analysis tool, they're motivated by their own sense of what code is crappy, they just know that their new OKR is to silence all the static analysis warnings. That's certainly a better outcome than not fixing anything, but that's still a worse outcome for the company, probably, than asking your devs what's wrong with the code and how they want to fix it.

fredophile 5y ago

I'd day there's another big group you're missing. People with large codebases that compile with tons of warnings that just get ignored. If you aren't willing to spend the time fixing the existing warnings to prevent problems why would you spend money on a tool that will give you more of them?