6 ms·

Semmle is basically datalog over source code. For what it works for, it works nice. But it is not a pancaea. Security vulnerability finding is almost certain

Semmle is basically datalog over source code. For what it works for, it works nice. But it is not a pancaea. Security vulnerability finding is almost certainly the wrong target for Semmle - I am unsure why they are trying to push that angle. There are much better stories in things like refactoring and understanding. (I say this having overseen a number of deployments for various reasons, some successful, some not)

sneak 7y ago
Nothing is a panacea. Things that help move the needle without requiring tons of time or effort are useful and valuable. I'm really glad to see more efforts in this area.
- DannyBee 7y ago
  While it's true that there is no pancaea, Semmle will not move the needle on vulnerability finding. This I have extensive data on. (I mean this in terms of capability, not sudden popularity) It would move the needle on a bunch else. It is a good tool for sure (and im very happy for them), i just think they will disappoint people by pressing this particular narrative, and wouldn't do so with a different narrative
  - intern4tional 7y ago
    I'd be curious as to why you think that. Are you able to provide more detail on that claim? I have extensive experience with Semmle and my experience drastically differs from you. With certain languages and a strong and diverse ruleset Semmle has it's strengths. In particular with native code (C, C++) and decent rules I have seen Semmle be very successful at finding certain classes of bugs.
  - lvh 7y ago
    The Datalog part is interesting! Do they have a bunch of rules to make graph queries work nicely, like Datomic pull syntax or maybe some pattern matching syntactic sugar? Is the underlying thing still an EAVT store? Is any of that information publicly available?
    - dantiberian 7y ago
      It looks like they have reasonable docs on their query language, in particular https://help.semmle.com/QL/learn-ql/about-ql.html#properties-of-ql https://help.semmle.com/QL/learn-ql/about-ql.html#properties... has some info on the QL language. https://help.semmle.com/lgtm-enterprise/user/help/generate-database.html https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revision, or snapshot, of the code.", though a triple store could qualify as relational here. I couldn't find much more than that about the implementation details though.
      - lvh 7y ago
        Right. I found those docs but they didn’t look like datalog queries at all. Of course that doesn’t mean they don’t compile down to datalog :)
  - lawnchair_larry 7y ago
    I would love to hear more about this data, as everyone I know who has used it for vulnerability finding has very good things to say. Semmle have also demonstrated its capabilities with some high profile examples.
    - tptacek 7y ago
      Who have you talked to about it? Outside Mozilla and Microsoft?
      - lawnchair_larry 7y ago
        I’d rather not name drop individuals or companies, partly because I don’t know that these entities want their business relationships public, but neither of the companies you named. I’ve also used it myself (you can too, at lgtm.com). I’m aware that MS is a customer, but I don’t think I’ve talked to anyone there about their experiences with it. As far as static analysis goes, which is inherently limited, it’s far better than anything else I’ve tried (which is most of them).
        tptacek 7y ago
        Fair! Thanks!
  - mphi 7y ago
    Could you share the details of your experience? My experience has been quite the opposite. I have been using Semmle daily to automate much of the vulnerability discovery process and I am extremely satisfied. We run it over millions of lines of Java code and have not yet run into scale or perf problems. Developing custom queries and defining security invariants in a logic language is, quite honestly, a joy.
- UncleMeat 7y ago
  Semmle does not scale, both in terms of their index design and their overall system design. This makes it poorly suited for truly global program properties and much better suited for things like refactoring.
  - muricula 7y ago
    It's being run frequently across the entire Windows OS repo. I have heard there is more work to be done to make it scale better, but it can scale.
    - intern4tional 7y ago
      Am Microsoft. Mountains were moved to make it scale, but that has been achieved. Semmle can scale with work - it just takes a lot of effort and code.
      - lawnchair_larry 7y ago
        What had to be done to make it scale?
        mynegation 7y ago
        I was not part of this effort but I did scale another static analysis tool for industrial size codebases and have a patent on it. To simplify a bit you can think of most static analysis algorithms in terms of graph problems where nodes are statements and functions and edges are flow of control and calls. On large codebases the amount of edges, nodes, and calculated data is just too big to keep in memory. The trick is to break the graph intelligently into parts, calculate some sort of summary information for each of them, distributing between cpus or computers, move up to the supegraph of graphs and perform higher level calculations on it.
      - DannyBee 7y ago
        Yeah, I think that is at least one of the issues. For us, for what it provides, it is not worth the time/effort vs just building our own tools or other options.
muricula 7y ago
I've seen coworkers run semmle queries across the entire Windows OS codebase and find hundreds of issues which were/could result in security vulnerabilities. They've also leveraged it for variant analysis. If I'm not mistaken, the security teams are the largest internal users of Semmle at Microsoft. You're right though, it's not a panacea, and it could probably be great for other uses too.
- wglb 7y ago
  Any idea what the false positive rate is?
  - tru3_power 7y ago
    If you use the out of the box rules for any of these tools the false positive rate will usually be pretty high. The trick is to write custom rules that are more tailored to your code.
    - wglb 7y ago
      So how much is involved in writing the rules, and at the end of it, what is the net false positive?
      - tru3_power 7y ago
        It took me a few months to get decent results with a low false positive rate. We haven’t had the tooling in place long enough to give hard stats but our aim is to have a false positive rate of less than 25%. Another great thing that these tools provide (if the results are valid) is that they let more junior members of the team/developers not as familiar with security issues to be able to understand the vulnerabilities found as they display a nice call flow graph/diagram that’s shows source to sink.
lawnchair_larry 7y ago
You’re wrong on that, security teams at the major tech companies love it, especially for variant analysis. Ask your coworkers at Google! One of which recently left to become Semmle’s Chief Security Officer.
- tptacek 7y ago
  First of all, Daniel Berlin is pretty senior at a reasonably large tech company a lot of us here have heard of. Secondly, I know Microsoft loves it, which is presumably where your telemetry comes from, and I know a lot of security people on Twitter are fans of the technology, but I've been asking around and "love it" is not the signal I'm getting from software security blue team people. "I installed it, I guess it does some stuff, we never think about it" is the modal feedback I've seen. I'm very interested in hearing success stories about this; the problem Semmle addresses is a huge part of the cost basis for my practice, and I'd love to hear that someone has gotten it working well.
  - lawnchair_larry 7y ago
    It’s absolutely useless in the wrong hands, so I don’t think you’d necessarily get a good signal by asking your average blue teamer. It’s a godsend for someone who spends a lot of time auditing code and has some experience writing code analysis tools. I mean think about it, if you wanted to write a query against the AST of a target, would you find that useful? Or in a given codebase, if you find one bug, would you like the ability to capture that in a query that can tell you if a similar mistake was made elsewhere? Out of the box, it isn’t going to give you much value. It’s the power of the query language, if it’s your job to do that, where you’ll see the benefits. But don’t take my word for it, just try it out. Their licensing model may be problematic for your use case though. I only vaguely understand what you do, but last I asked them about it, it’s not possible to get a personal license that a security person can use for multiple projects, and my read was that they had no interest in selling to individuals anytime soon.
    - tptacek 7y ago
      I mean, queries against an AST is sort of standard security tooling; the difference appears to be that Semmle (1) properly assigns types in C/C++ and (2) exports that query language. (1) makes sense to me; (2) I don't know how much better I'd get than just hand-writing tree walkers.
tru3_power 7y ago
I was looking at it earlier and the query syntax seems awesome. I’ve spent the last few months writing custom rules (queries) for fortify sca- which is another static code analysis tool and I must say, Semmle seems like it’s a lot easier to use. Static code analysis tooling is never the end all be all for vulnerability research, but it does let you express vulnerability patterns for implementation type vulnerabilities and find them at a mass scale (that is if your rules/queries are legit).
psygnisfive 7y ago
> Security vulnerability finding is almost certainly the wrong target for Semmle CVE-2019-5876 CVE-2019-16230 CVE-2019-16231 CVE-2019-16232 CVE-2019-16233 CVE-2019-16234 CVE-2019-15026 CVE-2019-14192 CVE-2019-14193 CVE-2019-14194 CVE-2019-14195 CVE-2019-14196 CVE-2019-14197 CVE-2019-14198 CVE-2019-14199 CVE-2019-14200 CVE-2019-14201 CVE-2019-14202 CVE-2019-14203 CVE-2019-14204 CVE-2019-14437 CVE-2019-14438 CVE-2019-14438 CVE-2019-14498 CVE-2019-14535 CVE-2019-14534 CVE-2019-14533 CVE-2019-14776 CVE-2019-14778 CVE-2019-14779 CVE-2019-14777 CVE-2019-14970 CVE-2019-15119 CVE-2019-14524 CVE-2019-14523 CVE-2019-7307 CVE-2019-11476 CVE-2019-13115 CVE-2019-3570 CVE-2019-13110 CVE-2019-13112 CVE-2019-13113 CVE-2019-13108 CVE-2019-13109 CVE-2019-13111 CVE-2019-13114 CVE-2019-3560 CVE-2019-9721 CVE-2019-9718 CVE-2019-9717 CVE-2019-9720 CVE-2019-9719 CVE-2018-20222 CVE-2019-3828 CVE-2019-6986 CVE-2019-5414 CVE-2018-4460 CVE-2018-16491 CVE-2018-16489 CVE-2018-16490 CVE-2018-19476 CVE-2018-19477 CVE-2018-19475 CVE-2018-19134 CVE-2018-16472 CVE-2018-18820 CVE-2018-4407 CVE-2018-16487 CVE-2018-4259 CVE-2018-4286 CVE-2018-4287 CVE-2018-4288 CVE-2018-4291 CVE-2019-5413 CVE-2018-16461 CVE-2018-16469 CVE-2018-16486 CVE-2018-16460 CVE-2018-16492 CVE-2018-11776 CVE-2018-8018 CVE-2018-8294 CVE-2018-4249 CVE-2018-8013 CVE-2018-5388 CVE-2018-1295 CVE-2018-4136 CVE-2018-4160 CVE-2018-1000140 CVE-2017-15692 CVE-2017-15693 CVE-2017-13904 CVE-2017-15089 CVE-2018-6834 CVE-2018-6835 CVE-2017-15713 CVE-2017-12634 CVE-2017-13782 CVE-2017-7545 CVE-2017-14949 CVE-2017-14868 CVE-2017-8046 CVE-2017-8045 CVE-2017-9805 CVE-2017-1000207 CVE-2017-1000208 CVE-2017-12612 CVE-2017-0141 https://lgtm.com/security/ https://lgtm.com/security/
- wglb 7y ago
  How would this list compare to other methods?
- DannyBee 7y ago
  Uh, I'm not sure why you believe this is an effective retort, perhaps you would like to explain?
  - captn3m0 7y ago
    >This list provides details about security vulnerabilities discovered by the Semmle Security Research Team using Semmle QL. Clearly, it works.

sneak 7y ago

Nothing is a panacea. Things that help move the needle without requiring tons of time or effort are useful and valuable. I'm really glad to see more efforts in this area.

DannyBee 7y ago
While it's true that there is no pancaea, Semmle will not move the needle on vulnerability finding. This I have extensive data on. (I mean this in terms of capability, not sudden popularity) It would move the needle on a bunch else. It is a good tool for sure (and im very happy for them), i just think they will disappoint people by pressing this particular narrative, and wouldn't do so with a different narrative
- intern4tional 7y ago
  I'd be curious as to why you think that. Are you able to provide more detail on that claim? I have extensive experience with Semmle and my experience drastically differs from you. With certain languages and a strong and diverse ruleset Semmle has it's strengths. In particular with native code (C, C++) and decent rules I have seen Semmle be very successful at finding certain classes of bugs.
- lvh 7y ago
  The Datalog part is interesting! Do they have a bunch of rules to make graph queries work nicely, like Datomic pull syntax or maybe some pattern matching syntactic sugar? Is the underlying thing still an EAVT store? Is any of that information publicly available?
  - dantiberian 7y ago
    It looks like they have reasonable docs on their query language, in particular https://help.semmle.com/QL/learn-ql/about-ql.html#properties-of-ql https://help.semmle.com/QL/learn-ql/about-ql.html#properties... has some info on the QL language. https://help.semmle.com/lgtm-enterprise/user/help/generate-database.html https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revision, or snapshot, of the code.", though a triple store could qualify as relational here. I couldn't find much more than that about the implementation details though.
    - lvh 7y ago
      Right. I found those docs but they didn’t look like datalog queries at all. Of course that doesn’t mean they don’t compile down to datalog :)
- lawnchair_larry 7y ago
  I would love to hear more about this data, as everyone I know who has used it for vulnerability finding has very good things to say. Semmle have also demonstrated its capabilities with some high profile examples.
  - tptacek 7y ago
    Who have you talked to about it? Outside Mozilla and Microsoft?
    - lawnchair_larry 7y ago
      I’d rather not name drop individuals or companies, partly because I don’t know that these entities want their business relationships public, but neither of the companies you named. I’ve also used it myself (you can too, at lgtm.com). I’m aware that MS is a customer, but I don’t think I’ve talked to anyone there about their experiences with it. As far as static analysis goes, which is inherently limited, it’s far better than anything else I’ve tried (which is most of them).
      - tptacek 7y ago
        Fair! Thanks!
- mphi 7y ago
  Could you share the details of your experience? My experience has been quite the opposite. I have been using Semmle daily to automate much of the vulnerability discovery process and I am extremely satisfied. We run it over millions of lines of Java code and have not yet run into scale or perf problems. Developing custom queries and defining security invariants in a logic language is, quite honestly, a joy.
UncleMeat 7y ago
Semmle does not scale, both in terms of their index design and their overall system design. This makes it poorly suited for truly global program properties and much better suited for things like refactoring.
- muricula 7y ago
  It's being run frequently across the entire Windows OS repo. I have heard there is more work to be done to make it scale better, but it can scale.
  - intern4tional 7y ago
    Am Microsoft. Mountains were moved to make it scale, but that has been achieved. Semmle can scale with work - it just takes a lot of effort and code.
    - lawnchair_larry 7y ago
      What had to be done to make it scale?
      - mynegation 7y ago
        I was not part of this effort but I did scale another static analysis tool for industrial size codebases and have a patent on it. To simplify a bit you can think of most static analysis algorithms in terms of graph problems where nodes are statements and functions and edges are flow of control and calls. On large codebases the amount of edges, nodes, and calculated data is just too big to keep in memory. The trick is to break the graph intelligently into parts, calculate some sort of summary information for each of them, distributing between cpus or computers, move up to the supegraph of graphs and perform higher level calculations on it.
    - DannyBee 7y ago
      Yeah, I think that is at least one of the issues. For us, for what it provides, it is not worth the time/effort vs just building our own tools or other options.

muricula 7y ago

I've seen coworkers run semmle queries across the entire Windows OS codebase and find hundreds of issues which were/could result in security vulnerabilities. They've also leveraged it for variant analysis. If I'm not mistaken, the security teams are the largest internal users of Semmle at Microsoft. You're right though, it's not a panacea, and it could probably be great for other uses too.

wglb 7y ago
Any idea what the false positive rate is?
- tru3_power 7y ago
  If you use the out of the box rules for any of these tools the false positive rate will usually be pretty high. The trick is to write custom rules that are more tailored to your code.
  - wglb 7y ago
    So how much is involved in writing the rules, and at the end of it, what is the net false positive?
    - tru3_power 7y ago
      It took me a few months to get decent results with a low false positive rate. We haven’t had the tooling in place long enough to give hard stats but our aim is to have a false positive rate of less than 25%. Another great thing that these tools provide (if the results are valid) is that they let more junior members of the team/developers not as familiar with security issues to be able to understand the vulnerabilities found as they display a nice call flow graph/diagram that’s shows source to sink.

lawnchair_larry 7y ago

You’re wrong on that, security teams at the major tech companies love it, especially for variant analysis. Ask your coworkers at Google! One of which recently left to become Semmle’s Chief Security Officer.

tptacek 7y ago
First of all, Daniel Berlin is pretty senior at a reasonably large tech company a lot of us here have heard of. Secondly, I know Microsoft loves it, which is presumably where your telemetry comes from, and I know a lot of security people on Twitter are fans of the technology, but I've been asking around and "love it" is not the signal I'm getting from software security blue team people. "I installed it, I guess it does some stuff, we never think about it" is the modal feedback I've seen. I'm very interested in hearing success stories about this; the problem Semmle addresses is a huge part of the cost basis for my practice, and I'd love to hear that someone has gotten it working well.
- lawnchair_larry 7y ago
  It’s absolutely useless in the wrong hands, so I don’t think you’d necessarily get a good signal by asking your average blue teamer. It’s a godsend for someone who spends a lot of time auditing code and has some experience writing code analysis tools. I mean think about it, if you wanted to write a query against the AST of a target, would you find that useful? Or in a given codebase, if you find one bug, would you like the ability to capture that in a query that can tell you if a similar mistake was made elsewhere? Out of the box, it isn’t going to give you much value. It’s the power of the query language, if it’s your job to do that, where you’ll see the benefits. But don’t take my word for it, just try it out. Their licensing model may be problematic for your use case though. I only vaguely understand what you do, but last I asked them about it, it’s not possible to get a personal license that a security person can use for multiple projects, and my read was that they had no interest in selling to individuals anytime soon.
  - tptacek 7y ago
    I mean, queries against an AST is sort of standard security tooling; the difference appears to be that Semmle (1) properly assigns types in C/C++ and (2) exports that query language. (1) makes sense to me; (2) I don't know how much better I'd get than just hand-writing tree walkers.

tru3_power 7y ago

I was looking at it earlier and the query syntax seems awesome. I’ve spent the last few months writing custom rules (queries) for fortify sca- which is another static code analysis tool and I must say, Semmle seems like it’s a lot easier to use. Static code analysis tooling is never the end all be all for vulnerability research, but it does let you express vulnerability patterns for implementation type vulnerabilities and find them at a mass scale (that is if your rules/queries are legit).

psygnisfive 7y ago

> Security vulnerability finding is almost certainly the wrong target for Semmle CVE-2019-5876 CVE-2019-16230 CVE-2019-16231 CVE-2019-16232 CVE-2019-16233 CVE-2019-16234 CVE-2019-15026 CVE-2019-14192 CVE-2019-14193 CVE-2019-14194 CVE-2019-14195 CVE-2019-14196 CVE-2019-14197 CVE-2019-14198 CVE-2019-14199 CVE-2019-14200 CVE-2019-14201 CVE-2019-14202 CVE-2019-14203 CVE-2019-14204 CVE-2019-14437 CVE-2019-14438 CVE-2019-14438 CVE-2019-14498 CVE-2019-14535 CVE-2019-14534 CVE-2019-14533 CVE-2019-14776 CVE-2019-14778 CVE-2019-14779 CVE-2019-14777 CVE-2019-14970 CVE-2019-15119 CVE-2019-14524 CVE-2019-14523 CVE-2019-7307 CVE-2019-11476 CVE-2019-13115 CVE-2019-3570 CVE-2019-13110 CVE-2019-13112 CVE-2019-13113 CVE-2019-13108 CVE-2019-13109 CVE-2019-13111 CVE-2019-13114 CVE-2019-3560 CVE-2019-9721 CVE-2019-9718 CVE-2019-9717 CVE-2019-9720 CVE-2019-9719 CVE-2018-20222 CVE-2019-3828 CVE-2019-6986 CVE-2019-5414 CVE-2018-4460 CVE-2018-16491 CVE-2018-16489 CVE-2018-16490 CVE-2018-19476 CVE-2018-19477 CVE-2018-19475 CVE-2018-19134 CVE-2018-16472 CVE-2018-18820 CVE-2018-4407 CVE-2018-16487 CVE-2018-4259 CVE-2018-4286 CVE-2018-4287 CVE-2018-4288 CVE-2018-4291 CVE-2019-5413 CVE-2018-16461 CVE-2018-16469 CVE-2018-16486 CVE-2018-16460 CVE-2018-16492 CVE-2018-11776 CVE-2018-8018 CVE-2018-8294 CVE-2018-4249 CVE-2018-8013 CVE-2018-5388 CVE-2018-1295 CVE-2018-4136 CVE-2018-4160 CVE-2018-1000140 CVE-2017-15692 CVE-2017-15693 CVE-2017-13904 CVE-2017-15089 CVE-2018-6834 CVE-2018-6835 CVE-2017-15713 CVE-2017-12634 CVE-2017-13782 CVE-2017-7545 CVE-2017-14949 CVE-2017-14868 CVE-2017-8046 CVE-2017-8045 CVE-2017-9805 CVE-2017-1000207 CVE-2017-1000208 CVE-2017-12612 CVE-2017-0141 https://lgtm.com/security/ https://lgtm.com/security/

wglb 7y ago
How would this list compare to other methods?
DannyBee 7y ago
Uh, I'm not sure why you believe this is an effective retort, perhaps you would like to explain?
- captn3m0 7y ago
  >This list provides details about security vulnerabilities discovered by the Semmle Security Research Team using Semmle QL. Clearly, it works.