7 ms·

The types of issues discovered (they mention null pointer access and resource and memory leaks) is much smaller than what a tool like Coverity will find (I use

by guepe 11y ago

The types of issues discovered (they mention null pointer access and resource and memory leaks) is much smaller than what a tool like Coverity will find (I use it). And they analyze C and Java, two languages supported by Coverity, a very mature tool... I am not certain of the proposed value, except it's free to other than Facebook - but not to Facebook, who pays engineers to develop this... Is this some kind of NIH syndrom by Facebook, or is there something I missed ?

Alupis 11y ago
Findbugs is also very good for Java apps, and is free. (Developed by the University of Maryland)[1] [1] http://findbugs.sourceforge.net/ http://findbugs.sourceforge.net/
- hbhakhra 11y ago
  These sorts of tools can get very annoying with their excessive warnings, but when they work, they save tons of time. We faced one bug at work where when multiple users were making a search at the same time, you would get bogus results back. With one user it would work consistently. We spent weeks tracking this downs and the problem: == instead of .equals() in Java The thing is, we were used to ignoring find bugs, but lo and behold it had pointed this problem out.
tptacek 11y ago
Isn't Coverity expensive?
- Locke1689 11y ago
  I would say a Coverity installation at Facebook is probably a "let's talk" level of expensive. That said, paying a team of expert engineers is also very expensive, not to mention the opportunity cost.
  - tptacek 11y ago
    Right, and for Facebook internally, whatever the number is, it's a speed bump. But some reasons not to use Coverity then: * Doing it in-house gives Facebook near total control over what the system is going to focus on; they can tailor it exactly to their problem set. * It's a worthwhile open source project, since most values of "expensive" mean "other projects won't ever use it". * If it gets any traction as an open source project, they can draft off the work other people will put into it.
    - Locke1689 11y ago
      I'd add * Facebook has recently hired a number of expert language theorists and practitioners. Doing it in-house 1) Gives them something to do 2) Serves to cement Facebook's language-expertise-brand recognition and dominance. The very fact that hiring is focusing on this group signals to me that this is an area which Facebook takes seriously and wants to be taken seriously in.
      - mschuster91 11y ago
        If Facebook ever goes down as a social network, they will without doubt still be a powerful technology company. FB has achieved a vertical integration in the IT world rivaled only by Google, Apple and maybe MS.
- justincormack 11y ago
  It is free for open source projects, but yes expensive otherwise.
Lewisham 11y ago
Coverity was evaluated a number of times at Google, and IIRC we decided it wasn't going to scale to the codebase size we needed it to. A separate and unrelated effort ended up with us building Tricorder [1]. Often perceived NIH at large companies for this sort of thing is simply a byproduct of scale that is unreasonable for external companies to have to worry about supporting. [1] http://research.google.com/pubs/pub43322.html http://research.google.com/pubs/pub43322.html
- ixtli 11y ago
  Your second point is very good; I get the feeling that NIH is simply a buzzword at the moment. Especially since a cursory glance Infer's source code will show that it was, in fact, invented somewhere else and purchased by Facebook.
pkaye 11y ago
Some of these tools (like Coverity) can be very comprehensive yet slow and expensive. We do our checking in layers like lint, memory check tools on every checkin and slower and expensive tools on a hourly/nightly basis.
Rezo 11y ago
Coverity is great, but for example on the mid-size service (10s but not 100s of kloc) that my team works on the analysis still takes hours. Therefore we only do it for prod releases, not on every commit or CI deployment. If you want to make static analysis part of the everyday development process, it has to be 1) very quick, ideally seconds; minutes at most 2) preferably something the developer can just run locally before pushing a change. If it's fast and easy enough, it simply becomes another code hygiene tool like a code formatter that you'll run continuously, perhaps even directly integrated into something like IntelliJ. To me Infer sounds like a nice complement to Coverity to catch issues as close to where they are introduced as possible. It might even be Good Enough for many projects to be the only tool, since Coverity is pretty expensive.
- cactusface 11y ago
  > If you want to make static analysis part of the everyday development process [...] Just to nitpick, I think your use of the term "static analysis" is a bit too broad. Every (or almost every) production compiler or JIT does static analysis intraprocedurally, that is, confined within a function / method / procedure. On the other hand, whole program / interprocedural static analysis quickly gets very expensive, usually because an alias / pointer analysis is involved, and that's what you need for null pointer checks and stuff. So I guess my point is, there is plenty of static analysis going on all the time, just not expensive whole program bug-finding analysis. Cheap bug-finding static analysis stuff is common, for example in GCC all those warning options to catch undefined behavior.
  - theblatte 11y ago
    (Infer dev here) One strength of Infer is that it is inter-procedural, yet not whole-program: each procedure gets analyzed independently. So it's cheap enough to run on large codebases while still able to find deep inter-procedural bugs.
    - cactusface 11y ago
      If you're analyzing procedures independently, why is it interprocedural? Interprocedural just means that you use some information about another procedure. This is expensive because if the information about one procedure changes during the analysis, you have to go and reanalyze all the dependent procedures. There are cheap but less accurate pointer analyses, is that why it's fast?
Cthulhu_ 11y ago
> Is this some kind of NIH syndrom by Facebook? Facebook employs thousands of developers and has a product that is pretty much done (or, IDK what they spend all their development budget on atm); they have the room to create new tools. TBF though, if this was a hobby project that wasn't linked to Facebook, it wouldn't get the attention it is getting right now.
xjia 11y ago
What is the underlying theory behind Coverity? Is it based on separation logic as well?
- buttproblem 11y ago
  I'd be interested in this as well; all I could find on their site is that it is uses "patened techniques." Considering the first seperation logic paper was published in 2001 [1] and Coverity founded in 2002 [2], they probably did not (at least originally) use the same techniques. I'm guessing their site is designed to sell to people and not have the details. [1] http://fbinfer.com/docs/separation-logic-and-bi-abduction.html#biabduction http://fbinfer.com/docs/separation-logic-and-bi-abduction.ht... [2] https://en.wikipedia.org/wiki/Coverity https://en.wikipedia.org/wiki/Coverity
cma 11y ago
how does the post positive break compare?
- cma 11y ago
  Sorry.. should have read "false positive rate"

by guepe 11y ago

Alupis 11y ago

Findbugs is also very good for Java apps, and is free. (Developed by the University of Maryland)[1] [1] http://findbugs.sourceforge.net/ http://findbugs.sourceforge.net/

hbhakhra 11y ago
These sorts of tools can get very annoying with their excessive warnings, but when they work, they save tons of time. We faced one bug at work where when multiple users were making a search at the same time, you would get bogus results back. With one user it would work consistently. We spent weeks tracking this downs and the problem: == instead of .equals() in Java The thing is, we were used to ignoring find bugs, but lo and behold it had pointed this problem out.

tptacek 11y ago

Isn't Coverity expensive?

Locke1689 11y ago
I would say a Coverity installation at Facebook is probably a "let's talk" level of expensive. That said, paying a team of expert engineers is also very expensive, not to mention the opportunity cost.
- tptacek 11y ago
  Right, and for Facebook internally, whatever the number is, it's a speed bump. But some reasons not to use Coverity then: * Doing it in-house gives Facebook near total control over what the system is going to focus on; they can tailor it exactly to their problem set. * It's a worthwhile open source project, since most values of "expensive" mean "other projects won't ever use it". * If it gets any traction as an open source project, they can draft off the work other people will put into it.
  - Locke1689 11y ago
    I'd add * Facebook has recently hired a number of expert language theorists and practitioners. Doing it in-house 1) Gives them something to do 2) Serves to cement Facebook's language-expertise-brand recognition and dominance. The very fact that hiring is focusing on this group signals to me that this is an area which Facebook takes seriously and wants to be taken seriously in.
    - mschuster91 11y ago
      If Facebook ever goes down as a social network, they will without doubt still be a powerful technology company. FB has achieved a vertical integration in the IT world rivaled only by Google, Apple and maybe MS.
justincormack 11y ago
It is free for open source projects, but yes expensive otherwise.

Lewisham 11y ago

Coverity was evaluated a number of times at Google, and IIRC we decided it wasn't going to scale to the codebase size we needed it to. A separate and unrelated effort ended up with us building Tricorder [1]. Often perceived NIH at large companies for this sort of thing is simply a byproduct of scale that is unreasonable for external companies to have to worry about supporting. [1] http://research.google.com/pubs/pub43322.html http://research.google.com/pubs/pub43322.html

ixtli 11y ago
Your second point is very good; I get the feeling that NIH is simply a buzzword at the moment. Especially since a cursory glance Infer's source code will show that it was, in fact, invented somewhere else and purchased by Facebook.

pkaye 11y ago

Some of these tools (like Coverity) can be very comprehensive yet slow and expensive. We do our checking in layers like lint, memory check tools on every checkin and slower and expensive tools on a hourly/nightly basis.

Rezo 11y ago

Coverity is great, but for example on the mid-size service (10s but not 100s of kloc) that my team works on the analysis still takes hours. Therefore we only do it for prod releases, not on every commit or CI deployment. If you want to make static analysis part of the everyday development process, it has to be 1) very quick, ideally seconds; minutes at most 2) preferably something the developer can just run locally before pushing a change. If it's fast and easy enough, it simply becomes another code hygiene tool like a code formatter that you'll run continuously, perhaps even directly integrated into something like IntelliJ. To me Infer sounds like a nice complement to Coverity to catch issues as close to where they are introduced as possible. It might even be Good Enough for many projects to be the only tool, since Coverity is pretty expensive.

cactusface 11y ago
> If you want to make static analysis part of the everyday development process [...] Just to nitpick, I think your use of the term "static analysis" is a bit too broad. Every (or almost every) production compiler or JIT does static analysis intraprocedurally, that is, confined within a function / method / procedure. On the other hand, whole program / interprocedural static analysis quickly gets very expensive, usually because an alias / pointer analysis is involved, and that's what you need for null pointer checks and stuff. So I guess my point is, there is plenty of static analysis going on all the time, just not expensive whole program bug-finding analysis. Cheap bug-finding static analysis stuff is common, for example in GCC all those warning options to catch undefined behavior.
- theblatte 11y ago
  (Infer dev here) One strength of Infer is that it is inter-procedural, yet not whole-program: each procedure gets analyzed independently. So it's cheap enough to run on large codebases while still able to find deep inter-procedural bugs.
  - cactusface 11y ago
    If you're analyzing procedures independently, why is it interprocedural? Interprocedural just means that you use some information about another procedure. This is expensive because if the information about one procedure changes during the analysis, you have to go and reanalyze all the dependent procedures. There are cheap but less accurate pointer analyses, is that why it's fast?

Cthulhu_ 11y ago

> Is this some kind of NIH syndrom by Facebook? Facebook employs thousands of developers and has a product that is pretty much done (or, IDK what they spend all their development budget on atm); they have the room to create new tools. TBF though, if this was a hobby project that wasn't linked to Facebook, it wouldn't get the attention it is getting right now.

xjia 11y ago

What is the underlying theory behind Coverity? Is it based on separation logic as well?

buttproblem 11y ago
I'd be interested in this as well; all I could find on their site is that it is uses "patened techniques." Considering the first seperation logic paper was published in 2001 [1] and Coverity founded in 2002 [2], they probably did not (at least originally) use the same techniques. I'm guessing their site is designed to sell to people and not have the details. [1] http://fbinfer.com/docs/separation-logic-and-bi-abduction.html#biabduction http://fbinfer.com/docs/separation-logic-and-bi-abduction.ht... [2] https://en.wikipedia.org/wiki/Coverity https://en.wikipedia.org/wiki/Coverity

cma 11y ago

how does the post positive break compare?

cma 11y ago
Sorry.. should have read "false positive rate"