2 ms·

I know Fermin, quite well, and I know the state of using it at Google. I technically am the one paying for the contract at this point! I've also met repeatedly

I know Fermin, quite well, and I know the state of using it at Google. I technically am the one paying for the contract at this point! I've also met repeatedly about it with all Google customers over the past few months, Both those currently using it, and those that stopped. Prior to that, I had met with most that used it but stopped, at the point they started using it. There were several large scale attempts/efforts to use semmle in various ways, by various teams. I'm trying to be as nice as possible here, since, as I said, I think it is a great tool for a lot of things, and I've been a strong supporter for this kind of technology for those cases (for years, in fact, as i'm sure some folks at Semmle can tell you) so I'd rather not burn it all down, which I expect is what would happen if i did a point by point explanation of everything bad about it. So I will instead reiterate my claim, and take my downvotes ;)

lawnchair_larry 7y ago
Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams use it in different ways, and it might be more suitable depending on your expectations. It’s also possible that the feedback I heard was during the honeymoon period, and practical issues outweigh the utility once you use it more. But I’ve seen real 0-days found with it, first hand and second hand, so I’m having trouble reconciling that with your account that it’s not useful for security.
- DannyBee 7y ago
  "and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later after i think about how to frame it :)
  - lawnchair_larry 7y ago
    So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback. If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
    - collingreene 7y ago
      Here is our experience building and using program analysis as part of our product security efforts at facebook: https://engineering.fb.com/security/zoncolan/ https://engineering.fb.com/security/zoncolan/. Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly. It finds about a third of the security vulns we unearth each year.
      - lawnchair_larry 7y ago
        That’s been my approach as well. An astonishingly large number of companies think they can buy an off the shelf static analysis tool and pipe the default output to developers. That’s counterproductive. A very small percentage of developers will understand the output, be able to assess the exploitability/severity, and care about fixing it. One might think you could then just have them take the “better safe than sorry” approach and fix everything, but FP rates for all of the commercial tools make that completely untenable. At the same time, you can’t expect to convince small teams of developers to model everything out and define sources/sinks using some obscure DSL that they have to learn. But, there are classes of issues that are extremely high impact, but only low accuracy static analysis rules can find the candidates. It’s that part in the middle that you don’t want to throw out, but you need security experts to vet. Other cases with high confidence checks are appropriate to short circuit straight to the devs, but it’s a bad first step.

lawnchair_larry 7y ago

Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams use it in different ways, and it might be more suitable depending on your expectations. It’s also possible that the feedback I heard was during the honeymoon period, and practical issues outweigh the utility once you use it more. But I’ve seen real 0-days found with it, first hand and second hand, so I’m having trouble reconciling that with your account that it’s not useful for security.

DannyBee 7y ago
"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later after i think about how to frame it :)
- lawnchair_larry 7y ago
  So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback. If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
  - collingreene 7y ago
    Here is our experience building and using program analysis as part of our product security efforts at facebook: https://engineering.fb.com/security/zoncolan/ https://engineering.fb.com/security/zoncolan/. Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly. It finds about a third of the security vulns we unearth each year.
    - lawnchair_larry 7y ago
      That’s been my approach as well. An astonishingly large number of companies think they can buy an off the shelf static analysis tool and pipe the default output to developers. That’s counterproductive. A very small percentage of developers will understand the output, be able to assess the exploitability/severity, and care about fixing it. One might think you could then just have them take the “better safe than sorry” approach and fix everything, but FP rates for all of the commercial tools make that completely untenable. At the same time, you can’t expect to convince small teams of developers to model everything out and define sources/sinks using some obscure DSL that they have to learn. But, there are classes of issues that are extremely high impact, but only low accuracy static analysis rules can find the candidates. It’s that part in the middle that you don’t want to throw out, but you need security experts to vet. Other cases with high confidence checks are appropriate to short circuit straight to the devs, but it’s a bad first step.