6 ms·

You’re wrong on that, security teams at the major tech companies love it, especially for variant analysis. Ask your coworkers at Google! One of which recently l

You’re wrong on that, security teams at the major tech companies love it, especially for variant analysis. Ask your coworkers at Google! One of which recently left to become Semmle’s Chief Security Officer.

tptacek 7y ago
First of all, Daniel Berlin is pretty senior at a reasonably large tech company a lot of us here have heard of. Secondly, I know Microsoft loves it, which is presumably where your telemetry comes from, and I know a lot of security people on Twitter are fans of the technology, but I've been asking around and "love it" is not the signal I'm getting from software security blue team people. "I installed it, I guess it does some stuff, we never think about it" is the modal feedback I've seen. I'm very interested in hearing success stories about this; the problem Semmle addresses is a huge part of the cost basis for my practice, and I'd love to hear that someone has gotten it working well.
- lawnchair_larry 7y ago
  It’s absolutely useless in the wrong hands, so I don’t think you’d necessarily get a good signal by asking your average blue teamer. It’s a godsend for someone who spends a lot of time auditing code and has some experience writing code analysis tools. I mean think about it, if you wanted to write a query against the AST of a target, would you find that useful? Or in a given codebase, if you find one bug, would you like the ability to capture that in a query that can tell you if a similar mistake was made elsewhere? Out of the box, it isn’t going to give you much value. It’s the power of the query language, if it’s your job to do that, where you’ll see the benefits. But don’t take my word for it, just try it out. Their licensing model may be problematic for your use case though. I only vaguely understand what you do, but last I asked them about it, it’s not possible to get a personal license that a security person can use for multiple projects, and my read was that they had no interest in selling to individuals anytime soon.
  - tptacek 7y ago
    I mean, queries against an AST is sort of standard security tooling; the difference appears to be that Semmle (1) properly assigns types in C/C++ and (2) exports that query language. (1) makes sense to me; (2) I don't know how much better I'd get than just hand-writing tree walkers.
    - apaprocki 7y ago
      Anecdata, I have also successfully used it to unearth complicated bugs that I already knew must exist, but didn’t have a good way to find. Think of a query like: Find all calls to function A that have an output pointer-pointer of type B in the last argument position and also have a boolean return type, verify the input dereferenced B is NULL, then verify that iff A assigns output to B that the return type is false and that the calling frame also includes a later call to function C to clean up B. This type of thing run on millions of lines of code. This can get as crazy as you want it to be and did work, but there is a “but.” The query language is so verbose and powerful, there were many, many ways to represent the same query that all had drastically different performance profiles. The docs were woefully underwhelming in that regard — they simply stated what a member did, but not anything at all related to memory or performance implications. That, coupled with the fact that nearly every complex query during dev hit the wall of the JVM killing it for taking too much time/mem, it became apparent that they must have tooling for the tooling to analyze performance and memory profiles of queries (not unlike all the effort put into SQL over the years). Also, no debugger; resorted to “datalog printf debugging” (ie, including internal clauses as outputs and chopping off later parts of the query...) I spent a few weeks only writing queries and in every non-trivial one I needed a senior person there to review/rearrange a few clauses to go from e.g. 30 minute runtime to 15 seconds. That left me with a feeling that I was constantly fighting the docs and lack of tooling and would constantly need their help to tune things. Nothing was fundamentally wrong with the queries — it was just not documented anywhere that some filtering should be done caller->down vs. other kinds of checks in the same query should be callee->up. I had questions regarding scaling up to 1B+ LoC like others in the thread but didn’t really get that far. It did successfully find the C/C++ bugs I was looking for once the queries had hours of investment from both sides, and we were also able to find bugs in a large custom JS codebase by mocking all the things it would need to understand to eval the code. Whether or not that investment makes sense is an individual/team/org question, but if they ever seriously invest in a query debugger and profiling, they’ll be pretty hard to beat.
DannyBee 7y ago
I know Fermin, quite well, and I know the state of using it at Google. I technically am the one paying for the contract at this point! I've also met repeatedly about it with all Google customers over the past few months, Both those currently using it, and those that stopped. Prior to that, I had met with most that used it but stopped, at the point they started using it. There were several large scale attempts/efforts to use semmle in various ways, by various teams. I'm trying to be as nice as possible here, since, as I said, I think it is a great tool for a lot of things, and I've been a strong supporter for this kind of technology for those cases (for years, in fact, as i'm sure some folks at Semmle can tell you) so I'd rather not burn it all down, which I expect is what would happen if i did a point by point explanation of everything bad about it. So I will instead reiterate my claim, and take my downvotes ;)
- lawnchair_larry 7y ago
  Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams use it in different ways, and it might be more suitable depending on your expectations. It’s also possible that the feedback I heard was during the honeymoon period, and practical issues outweigh the utility once you use it more. But I’ve seen real 0-days found with it, first hand and second hand, so I’m having trouble reconciling that with your account that it’s not useful for security.
  - DannyBee 7y ago
    "and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later after i think about how to frame it :)

tptacek 7y ago

First of all, Daniel Berlin is pretty senior at a reasonably large tech company a lot of us here have heard of. Secondly, I know Microsoft loves it, which is presumably where your telemetry comes from, and I know a lot of security people on Twitter are fans of the technology, but I've been asking around and "love it" is not the signal I'm getting from software security blue team people. "I installed it, I guess it does some stuff, we never think about it" is the modal feedback I've seen. I'm very interested in hearing success stories about this; the problem Semmle addresses is a huge part of the cost basis for my practice, and I'd love to hear that someone has gotten it working well.

lawnchair_larry 7y ago
It’s absolutely useless in the wrong hands, so I don’t think you’d necessarily get a good signal by asking your average blue teamer. It’s a godsend for someone who spends a lot of time auditing code and has some experience writing code analysis tools. I mean think about it, if you wanted to write a query against the AST of a target, would you find that useful? Or in a given codebase, if you find one bug, would you like the ability to capture that in a query that can tell you if a similar mistake was made elsewhere? Out of the box, it isn’t going to give you much value. It’s the power of the query language, if it’s your job to do that, where you’ll see the benefits. But don’t take my word for it, just try it out. Their licensing model may be problematic for your use case though. I only vaguely understand what you do, but last I asked them about it, it’s not possible to get a personal license that a security person can use for multiple projects, and my read was that they had no interest in selling to individuals anytime soon.
- tptacek 7y ago
  I mean, queries against an AST is sort of standard security tooling; the difference appears to be that Semmle (1) properly assigns types in C/C++ and (2) exports that query language. (1) makes sense to me; (2) I don't know how much better I'd get than just hand-writing tree walkers.
  - apaprocki 7y ago
    Anecdata, I have also successfully used it to unearth complicated bugs that I already knew must exist, but didn’t have a good way to find. Think of a query like: Find all calls to function A that have an output pointer-pointer of type B in the last argument position and also have a boolean return type, verify the input dereferenced B is NULL, then verify that iff A assigns output to B that the return type is false and that the calling frame also includes a later call to function C to clean up B. This type of thing run on millions of lines of code. This can get as crazy as you want it to be and did work, but there is a “but.” The query language is so verbose and powerful, there were many, many ways to represent the same query that all had drastically different performance profiles. The docs were woefully underwhelming in that regard — they simply stated what a member did, but not anything at all related to memory or performance implications. That, coupled with the fact that nearly every complex query during dev hit the wall of the JVM killing it for taking too much time/mem, it became apparent that they must have tooling for the tooling to analyze performance and memory profiles of queries (not unlike all the effort put into SQL over the years). Also, no debugger; resorted to “datalog printf debugging” (ie, including internal clauses as outputs and chopping off later parts of the query...) I spent a few weeks only writing queries and in every non-trivial one I needed a senior person there to review/rearrange a few clauses to go from e.g. 30 minute runtime to 15 seconds. That left me with a feeling that I was constantly fighting the docs and lack of tooling and would constantly need their help to tune things. Nothing was fundamentally wrong with the queries — it was just not documented anywhere that some filtering should be done caller->down vs. other kinds of checks in the same query should be callee->up. I had questions regarding scaling up to 1B+ LoC like others in the thread but didn’t really get that far. It did successfully find the C/C++ bugs I was looking for once the queries had hours of investment from both sides, and we were also able to find bugs in a large custom JS codebase by mocking all the things it would need to understand to eval the code. Whether or not that investment makes sense is an individual/team/org question, but if they ever seriously invest in a query debugger and profiling, they’ll be pretty hard to beat.

DannyBee 7y ago

I know Fermin, quite well, and I know the state of using it at Google. I technically am the one paying for the contract at this point! I've also met repeatedly about it with all Google customers over the past few months, Both those currently using it, and those that stopped. Prior to that, I had met with most that used it but stopped, at the point they started using it. There were several large scale attempts/efforts to use semmle in various ways, by various teams. I'm trying to be as nice as possible here, since, as I said, I think it is a great tool for a lot of things, and I've been a strong supporter for this kind of technology for those cases (for years, in fact, as i'm sure some folks at Semmle can tell you) so I'd rather not burn it all down, which I expect is what would happen if i did a point by point explanation of everything bad about it. So I will instead reiterate my claim, and take my downvotes ;)

lawnchair_larry 7y ago
Interesting - I’d like your unfiltered take, but I totally understand your position. I’ve heard very positive feedback from one security person at Google, and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it. If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams use it in different ways, and it might be more suitable depending on your expectations. It’s also possible that the feedback I heard was during the honeymoon period, and practical issues outweigh the utility once you use it more. But I’ve seen real 0-days found with it, first hand and second hand, so I’m having trouble reconciling that with your account that it’s not useful for security.
- DannyBee 7y ago
  "and I don’t know Fermin, but taking a CSO role there would suggest to me that he believes in it." Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :) I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation. IE it's not the kind of offer that really required a lot of faith I'll try to write a bit more later after i think about how to frame it :)