3 ms·

While it's true that there is no pancaea, Semmle will not move the needle on vulnerability finding. This I have extensive data on. (I mean this in terms of ca

While it's true that there is no pancaea, Semmle will not move the needle on vulnerability finding. This I have extensive data on. (I mean this in terms of capability, not sudden popularity) It would move the needle on a bunch else. It is a good tool for sure (and im very happy for them), i just think they will disappoint people by pressing this particular narrative, and wouldn't do so with a different narrative

intern4tional 7y ago
I'd be curious as to why you think that. Are you able to provide more detail on that claim? I have extensive experience with Semmle and my experience drastically differs from you. With certain languages and a strong and diverse ruleset Semmle has it's strengths. In particular with native code (C, C++) and decent rules I have seen Semmle be very successful at finding certain classes of bugs.
lvh 7y ago
The Datalog part is interesting! Do they have a bunch of rules to make graph queries work nicely, like Datomic pull syntax or maybe some pattern matching syntactic sugar? Is the underlying thing still an EAVT store? Is any of that information publicly available?
- dantiberian 7y ago
  It looks like they have reasonable docs on their query language, in particular https://help.semmle.com/QL/learn-ql/about-ql.html#properties-of-ql https://help.semmle.com/QL/learn-ql/about-ql.html#properties... has some info on the QL language. https://help.semmle.com/lgtm-enterprise/user/help/generate-database.html https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revision, or snapshot, of the code.", though a triple store could qualify as relational here. I couldn't find much more than that about the implementation details though.
  - lvh 7y ago
    Right. I found those docs but they didn’t look like datalog queries at all. Of course that doesn’t mean they don’t compile down to datalog :)
lawnchair_larry 7y ago
I would love to hear more about this data, as everyone I know who has used it for vulnerability finding has very good things to say. Semmle have also demonstrated its capabilities with some high profile examples.
- tptacek 7y ago
  Who have you talked to about it? Outside Mozilla and Microsoft?
  - lawnchair_larry 7y ago
    I’d rather not name drop individuals or companies, partly because I don’t know that these entities want their business relationships public, but neither of the companies you named. I’ve also used it myself (you can too, at lgtm.com). I’m aware that MS is a customer, but I don’t think I’ve talked to anyone there about their experiences with it. As far as static analysis goes, which is inherently limited, it’s far better than anything else I’ve tried (which is most of them).
    - tptacek 7y ago
      Fair! Thanks!
mphi 7y ago
Could you share the details of your experience? My experience has been quite the opposite. I have been using Semmle daily to automate much of the vulnerability discovery process and I am extremely satisfied. We run it over millions of lines of Java code and have not yet run into scale or perf problems. Developing custom queries and defining security invariants in a logic language is, quite honestly, a joy.

intern4tional 7y ago

I'd be curious as to why you think that. Are you able to provide more detail on that claim? I have extensive experience with Semmle and my experience drastically differs from you. With certain languages and a strong and diverse ruleset Semmle has it's strengths. In particular with native code (C, C++) and decent rules I have seen Semmle be very successful at finding certain classes of bugs.

lvh 7y ago

The Datalog part is interesting! Do they have a bunch of rules to make graph queries work nicely, like Datomic pull syntax or maybe some pattern matching syntactic sugar? Is the underlying thing still an EAVT store? Is any of that information publicly available?

dantiberian 7y ago
It looks like they have reasonable docs on their query language, in particular https://help.semmle.com/QL/learn-ql/about-ql.html#properties-of-ql https://help.semmle.com/QL/learn-ql/about-ql.html#properties... has some info on the QL language. https://help.semmle.com/lgtm-enterprise/user/help/generate-database.html https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revision, or snapshot, of the code.", though a triple store could qualify as relational here. I couldn't find much more than that about the implementation details though.
- lvh 7y ago
  Right. I found those docs but they didn’t look like datalog queries at all. Of course that doesn’t mean they don’t compile down to datalog :)

lawnchair_larry 7y ago

I would love to hear more about this data, as everyone I know who has used it for vulnerability finding has very good things to say. Semmle have also demonstrated its capabilities with some high profile examples.

tptacek 7y ago
Who have you talked to about it? Outside Mozilla and Microsoft?
- lawnchair_larry 7y ago
  I’d rather not name drop individuals or companies, partly because I don’t know that these entities want their business relationships public, but neither of the companies you named. I’ve also used it myself (you can too, at lgtm.com). I’m aware that MS is a customer, but I don’t think I’ve talked to anyone there about their experiences with it. As far as static analysis goes, which is inherently limited, it’s far better than anything else I’ve tried (which is most of them).
  - tptacek 7y ago
    Fair! Thanks!

mphi 7y ago

Could you share the details of your experience? My experience has been quite the opposite. I have been using Semmle daily to automate much of the vulnerability discovery process and I am extremely satisfied. We run it over millions of lines of Java code and have not yet run into scale or perf problems. Developing custom queries and defining security invariants in a logic language is, quite honestly, a joy.