4 ms·

Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I

Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams use it in different ways, and it might be more suitable depending on your expectations. It’s also possible that the feedback I heard was during the honeymoon period, and practical issues outweigh the utility once you use it more. But I’ve seen real 0-days found with it, first hand and second hand, so I’m having trouble reconciling that with your account that it’s not useful for security.

DannyBee 7y ago
"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later after i think about how to frame it :)
- lawnchair_larry 7y ago
  So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback. If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
  - collingreene 7y ago
    Here is our experience building and using program analysis as part of our product security efforts at facebook: https://engineering.fb.com/security/zoncolan/ https://engineering.fb.com/security/zoncolan/. Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly. It finds about a third of the security vulns we unearth each year.
    - lawnchair_larry 7y ago
      That’s been my approach as well. An astonishingly large number of companies think they can buy an off the shelf static analysis tool and pipe the default output to developers. That’s counterproductive. A very small percentage of developers will understand the output, be able to assess the exploitability/severity, and care about fixing it. One might think you could then just have them take the “better safe than sorry” approach and fix everything, but FP rates for all of the commercial tools make that completely untenable. At the same time, you can’t expect to convince small teams of developers to model everything out and define sources/sinks using some obscure DSL that they have to learn. But, there are classes of issues that are extremely high impact, but only low accuracy static analysis rules can find the candidates. It’s that part in the middle that you don’t want to throw out, but you need security experts to vet. Other cases with high confidence checks are appropriate to short circuit straight to the devs, but it’s a bad first step.
  - deleted 7y ago
    [deleted]
    - jlokier 7y ago
      I think there's some great feedback here, for anyone at Semmle thinking about how to develop the tool further.
      - 7y ago
- fjserna 7y ago
  Fermin here. Danny, I loved working with you at Google but let me correct you about my promotion and money. Not true. I respect your point of view around our technology. You may like it or not (some folks love it), but please do not make statements about me you do not really know :) And to be clear, I believe this technology makes security researchers scale on different aspects. At least I had first hand experience with this and our goal is to make security easy for non security folks... this technology enables us to do this. Happy to sync in private over a coffee!
  - deleted 7y ago
    [deleted]
    - fjserna 7y ago
      All good Danny, we had good times at Google... let's remember those :) Coffee offer is still there!
- oegerikus 7y ago
  Danny, I am the CEO and founder of Semmle. I will refrain from arguing about the value of our product and technology. However, I must correct your statement about Fermín which is utterly false. He took a huge pay cut to come to Semmle. Please stick to facts when talking about people.
  - deleted 7y ago
    [deleted]

DannyBee 7y ago

"and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later after i think about how to frame it :)

lawnchair_larry 7y ago
So I did read a whitepaper about static analysis at Google, and how it was largely self-serve - let developers run the tools and fix what it tells them to as they see fit. I’m wondering if it was under this model where you found it was not useful. I would not expect it to provide much value in that scenario, and would not be surprised by your feedback. If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
- collingreene 7y ago
  Here is our experience building and using program analysis as part of our product security efforts at facebook: https://engineering.fb.com/security/zoncolan/ https://engineering.fb.com/security/zoncolan/. Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly. It finds about a third of the security vulns we unearth each year.
  - lawnchair_larry 7y ago
    That’s been my approach as well. An astonishingly large number of companies think they can buy an off the shelf static analysis tool and pipe the default output to developers. That’s counterproductive. A very small percentage of developers will understand the output, be able to assess the exploitability/severity, and care about fixing it. One might think you could then just have them take the “better safe than sorry” approach and fix everything, but FP rates for all of the commercial tools make that completely untenable. At the same time, you can’t expect to convince small teams of developers to model everything out and define sources/sinks using some obscure DSL that they have to learn. But, there are classes of issues that are extremely high impact, but only low accuracy static analysis rules can find the candidates. It’s that part in the middle that you don’t want to throw out, but you need security experts to vet. Other cases with high confidence checks are appropriate to short circuit straight to the devs, but it’s a bad first step.
- deleted 7y ago
  [deleted]
  - jlokier 7y ago
    I think there's some great feedback here, for anyone at Semmle thinking about how to develop the tool further.
    - 7y ago
fjserna 7y ago
Fermin here. Danny, I loved working with you at Google but let me correct you about my promotion and money. Not true. I respect your point of view around our technology. You may like it or not (some folks love it), but please do not make statements about me you do not really know :) And to be clear, I believe this technology makes security researchers scale on different aspects. At least I had first hand experience with this and our goal is to make security easy for non security folks... this technology enables us to do this. Happy to sync in private over a coffee!
- deleted 7y ago
  [deleted]
  - fjserna 7y ago
    All good Danny, we had good times at Google... let's remember those :) Coffee offer is still there!
oegerikus 7y ago
Danny, I am the CEO and founder of Semmle. I will refrain from arguing about the value of our product and technology. However, I must correct your statement about Fermín which is utterly false. He took a huge pay cut to come to Semmle. Please stick to facts when talking about people.
- deleted 7y ago
  [deleted]