4 ms·
Nothing is a panacea. Things that help move the needle without requiring tons of time or effort are useful and valuable. I'm really glad to see more efforts i
by sneak 7y ago
Nothing is a panacea. Things that help move the needle without requiring tons of time or effort are useful and valuable. I'm really glad to see more efforts in this area.
- DannyBee 7y agoWhile it's true that there is no pancaea, Semmle will not move the needle on vulnerability finding. This I have extensive data on. (I mean this in terms of capability, not sudden popularity) It would move the needle on a bunch else. It is a good tool for sure (and im very happy for them), i just think they will disappoint people by pressing this particular narrative, and wouldn't do so with a different narrative
- intern4tional 7y agoI'd be curious as to why you think that. Are you able to provide more detail on that claim? I have extensive experience with Semmle and my experience drastically differs from you. With certain languages and a strong and diverse ruleset Semmle has it's strengths. In particular with native code (C, C++) and decent rules I have seen Semmle be very successful at finding certain classes of bugs.
- lvh 7y agoThe Datalog part is interesting! Do they have a bunch of rules to make graph queries work nicely, like Datomic pull syntax or maybe some pattern matching syntactic sugar? Is the underlying thing still an EAVT store? Is any of that information publicly available?
- dantiberian 7y agoIt looks like they have reasonable docs on their query language, in particular https://help.semmle.com/QL/learn-ql/about-ql.html#properties-of-ql https://help.semmle.com/QL/learn-ql/about-ql.html#properties... has some info on the QL language. https://help.semmle.com/lgtm-enterprise/user/help/generate-database.html https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revision, or snapshot, of the code.", though a triple store could qualify as relational here. I couldn't find much more than that about the implementation details though.
- lvh 7y agoRight. I found those docs but they didn’t look like datalog queries at all. Of course that doesn’t mean they don’t compile down to datalog :)
- lawnchair_larry 7y agoI would love to hear more about this data, as everyone I know who has used it for vulnerability finding has very good things to say. Semmle have also demonstrated its capabilities with some high profile examples.
- tptacek 7y agoWho have you talked to about it? Outside Mozilla and Microsoft?
- lawnchair_larry 7y agoI’d rather not name drop individuals or companies, partly because I don’t know that these entities want their business relationships public, but neither of the companies you named. I’ve also used it myself (you can too, at lgtm.com). I’m aware that MS is a customer, but I don’t think I’ve talked to anyone there about their experiences with it. As far as static analysis goes, which is inherently limited, it’s far better than anything else I’ve tried (which is most of them).
- tptacek 7y agoFair! Thanks!
- mphi 7y agoCould you share the details of your experience? My experience has been quite the opposite. I have been using Semmle daily to automate much of the vulnerability discovery process and I am extremely satisfied. We run it over millions of lines of Java code and have not yet run into scale or perf problems. Developing custom queries and defining security invariants in a logic language is, quite honestly, a joy.
- UncleMeat 7y agoSemmle does not scale, both in terms of their index design and their overall system design. This makes it poorly suited for truly global program properties and much better suited for things like refactoring.
- muricula 7y agoIt's being run frequently across the entire Windows OS repo. I have heard there is more work to be done to make it scale better, but it can scale.
- intern4tional 7y agoAm Microsoft. Mountains were moved to make it scale, but that has been achieved. Semmle can scale with work - it just takes a lot of effort and code.
- lawnchair_larry 7y agoWhat had to be done to make it scale?
- mynegation 7y agoI was not part of this effort but I did scale another static analysis tool for industrial size codebases and have a patent on it. To simplify a bit you can think of most static analysis algorithms in terms of graph problems where nodes are statements and functions and edges are flow of control and calls. On large codebases the amount of edges, nodes, and calculated data is just too big to keep in memory. The trick is to break the graph intelligently into parts, calculate some sort of summary information for each of them, distributing between cpus or computers, move up to the supegraph of graphs and perform higher level calculations on it.
- DannyBee 7y agoYeah, I think that is at least one of the issues. For us, for what it provides, it is not worth the time/effort vs just building our own tools or other options.