6 ms·

The article cites awesome stats about the effectiveness of showing the results of static analysis at "diff time" (in code review) vs. outside the dev workflow.

by sqs 7y ago

The article cites awesome stats about the effectiveness of showing the results of static analysis at "diff time" (in code review) vs. outside the dev workflow. > For classes of bugs intended for all or a wide variety of engineers on a given platform, we have gravitated toward a "diff time" deployment, where analyzers participate as bots in code review, making automatic comments when an engineer submits a code modification. Later, we recount a striking situation where the diff time deployment saw a 70% fix rate, where a more traditional "offline" or "batch" deployment (where bug lists are presented to engineers, outside their workflow) saw a 0% fix rate. > ... > The response was stunning: we were greeted by near silence. We assigned 20-30 issues to developers, and almost none of them were acted on. We had worked hard to get the false positive rate down to what we thought was less than 20%, and yet the fix rate—the proportion of reported issues that developers resolved—was near zero. > Next, we switched Infer on at diff time. The response of engineers was just as stunning: the fix rate rocketed to over 70%. The same program analysis, with same false positive rate, had much greater impact when deployed at diff time. I am the Sourcegraph CEO, and we've long been aware of tools like this at Facebook, and Google's similar tool (described in another ACM article, https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...). We are working on bringing static analysis into the dev workflow, and letting you run campaigns to apply static analysis fixes across your entire codebase. Check out https://about.sourcegraph.com/product/automation/ https://about.sourcegraph.com/product/automation/ for info and screencasts to see what this looks like.

tempguy9999 7y ago
I've read something very similar before (also facebook) and one thing disturbed me, and it's given in your quote, the difference in bugfix uptakes by devs depending on how it's presented. I have to say that shocked me as it appears so unprofessional (yes, I read the reasons but still). The root of that shock was the human factor. Us devs aren't machine parts, but our essential job is to do stuff and fix the inevitable bugs. The fact that presentation of bug reports could change things so radically was an eye opener, and one I'll generalise and remember. Then again I've never worked on huge codebases, and I assume the devs there were top engineers so I'm not attributing this to laziness or indifference - but still, amazing and not in a positive way! I'd just assumed static analysis = bugs found = bugs fixed = better code; end of story.
- mirceal 7y ago
  human factor but also incentives. if someone comes to you with a bazillion issues that you cannot possibly fix in the X hours you have during the day what are you going to do? you're going to prioritize and you're going to fix the most important bugs, right? if you're developing a new feature and things are pointed out during the code review (bots or no bots) what are you going to do? fix it, right? you want to ship whatever you're doing with high quality. so the problem outlined below is a false one. you can never fix all the possible bugs if the codebase if nothing but trivial.
  - tempguy9999 7y ago
    > you're going to prioritize and you're going to fix the most important bugs, right? That doesn't make sense. Presented one way, the bugs were fixed to a 70% rate. Presented another way, 0%. Presumably the bugs in each case were about equivalent in their severity (else the conclusion is meaningless) so severity was not the driving factor in what got fixed. > you can never fix all the possible bugs if the codebase if nothing but trivial. That's a non-sequitur. No-one suggested they could fix all, or should try. It was about known bugs presented to them.
    - wffurr 7y ago
      It makes perfect sense. If you file a bug assigned to me that doesn't seem to actually affect users or isn't aligned with the work I am already committed too, it goes in the backlog and I may or may not ever get to it. If it's a bot that generated the bug, it comes after user reports, and I may never get to all of those. If it's a comment on a code review for code I am already working on, then of course I want my new or changed code to be up to date with standards and avoid error prone patterns, etc.
      - tempguy9999 7y ago
        @mirceal, @wffurr, I see what you're saying and you're both right. However that does leave a major question for me; if devs are triaging their own bugs to this extent (70% vs 0% fix), is this safe, because those bugs will relate ultimately to business requirements related to uptime and correctness.
tyingq 7y ago
We are seeing similar with SonarQube[1]. Putting it in the CI pipeline encourages iterative progress. It's much less overwhelming than a periodic batch scan that outputs a ton of mixed positive/false-positive issues. It's not perfect, but the smaller scale reporting is easier to deal with, fix, add suppression rules, etc. Sounds obvious, I suppose. [1] https://www.sonarqube.org/ https://www.sonarqube.org/

by sqs 7y ago

tempguy9999 7y ago

I've read something very similar before (also facebook) and one thing disturbed me, and it's given in your quote, the difference in bugfix uptakes by devs depending on how it's presented. I have to say that shocked me as it appears so unprofessional (yes, I read the reasons but still). The root of that shock was the human factor. Us devs aren't machine parts, but our essential job is to do stuff and fix the inevitable bugs. The fact that presentation of bug reports could change things so radically was an eye opener, and one I'll generalise and remember. Then again I've never worked on huge codebases, and I assume the devs there were top engineers so I'm not attributing this to laziness or indifference - but still, amazing and not in a positive way! I'd just assumed static analysis = bugs found = bugs fixed = better code; end of story.

mirceal 7y ago
human factor but also incentives. if someone comes to you with a bazillion issues that you cannot possibly fix in the X hours you have during the day what are you going to do? you're going to prioritize and you're going to fix the most important bugs, right? if you're developing a new feature and things are pointed out during the code review (bots or no bots) what are you going to do? fix it, right? you want to ship whatever you're doing with high quality. so the problem outlined below is a false one. you can never fix all the possible bugs if the codebase if nothing but trivial.
- tempguy9999 7y ago
  > you're going to prioritize and you're going to fix the most important bugs, right? That doesn't make sense. Presented one way, the bugs were fixed to a 70% rate. Presented another way, 0%. Presumably the bugs in each case were about equivalent in their severity (else the conclusion is meaningless) so severity was not the driving factor in what got fixed. > you can never fix all the possible bugs if the codebase if nothing but trivial. That's a non-sequitur. No-one suggested they could fix all, or should try. It was about known bugs presented to them.
  - wffurr 7y ago
    It makes perfect sense. If you file a bug assigned to me that doesn't seem to actually affect users or isn't aligned with the work I am already committed too, it goes in the backlog and I may or may not ever get to it. If it's a bot that generated the bug, it comes after user reports, and I may never get to all of those. If it's a comment on a code review for code I am already working on, then of course I want my new or changed code to be up to date with standards and avoid error prone patterns, etc.
    - tempguy9999 7y ago
      @mirceal, @wffurr, I see what you're saying and you're both right. However that does leave a major question for me; if devs are triaging their own bugs to this extent (70% vs 0% fix), is this safe, because those bugs will relate ultimately to business requirements related to uptime and correctness.

tyingq 7y ago

We are seeing similar with SonarQube[1]. Putting it in the CI pipeline encourages iterative progress. It's much less overwhelming than a periodic batch scan that outputs a ton of mixed positive/false-positive issues. It's not perfect, but the smaller scale reporting is easier to deal with, fix, add suppression rules, etc. Sounds obvious, I suppose. [1] https://www.sonarqube.org/ https://www.sonarqube.org/