4 ms·

Hey HN, I’m the author of this post and a contributor to Semgrep. Happy to answer questions and hear feedback! I’m excited to try to lower the barrier to writin

by ievans 6y ago

Hey HN, I’m the author of this post and a contributor to Semgrep. Happy to answer questions and hear feedback! I’m excited to try to lower the barrier to writing a simple lint (or more complex program analysis) that previously only a static analysis expert could do; we’ve gotten contributions from people who don’t know what an abstract syntax tree is! The userbase for Semgrep is almost evenly split between security engineers using it for hunting/enforcement and developers looking for bugs; we’ve tried to collect examples for both use cases at https://semgrep.dev/explore https://semgrep.dev/explore.

carlmr 6y ago
First of all, I love the idea of semgrep, but can't use it since we're using C++. Is there any chance for C++ support in the future?
- lvh 6y ago
  Not GP, but one the one hand semgrep has a real honest to goodness parser at its core; on the other hand I'd expect C++ to have sufficiently complicated semantics that it needs some understanding of C++ specific mechanics to be useful. Furthermore, you'd need preprocessor and template expansion magic to really get to the bottom of it. Effectively this is the same problem e.g. javacpp has.
- ievans 6y ago
  The good news is that we’ve replaced almost all the homegrown parsers that were written while the tool was at Facebook and we’re using the now tree-sitter project, which already has parsers for 40+ languages. There is a tree-sitter-cpp project we can and will eventually integrate! The bad news is this requires the code to not use heavily macros to be parseable as-is. So really the difficulty is not C++ but rather the pre-processor.
  - carlmr 6y ago
    That sounds really cool, would it be possible to ignore the macros or can the code not be parsed at all then?
  - philsnow 6y ago
    I guess you could run the preprocessor and then run tree-sitter-cpp / semgrep on the preprocessed output, but the problem would then be trying to tie any findings from that to the original source. Do gcc/clang/any other preprocessor create "source maps" that could facilitate that? GCC looks like it has a `-fdebug-cpp` that "[...] dumps debugging information about location maps. Every token in the output is preceded by the dump of the map its location belongs to."
    - carlmr 6y ago
      The preprocessed output of GCC and Clang usually contains the file names.
      - inetknght 6y ago
        Better than that, compilers can be told to include line numbers too. This thread would love to learn about Compiler Explorer https://gcc.godbolt.org/ https://gcc.godbolt.org/ which works for C++ and many other languages.
        philsnow 6y ago
        Line numbers are a good start, but JS source maps go from output source byte ranges to input source byte ranges. I don't write or even read a lot of C++ these days but I recall from when I did that a major pain point was deciphering compiler warnings/errors when there are a lot of templates, macros, or both. Seems like the problem has been around forever.
    - lvh 6y ago
      Right; but this turns out to be pretty tricky in practice. I've attempted to do this for even relatively straightforward code (libsodium--complex in implementation, though not in API) with libclang and it was not particularly pleasant. Some prior art for reference: https://github.com/bytedeco/javacpp/issues/51 https://github.com/bytedeco/javacpp/issues/51
dti 6y ago
Is Semmle, offering CodeQL language and LGTM service, and recently acquired by Github, doing a similar thing (https://semmle.com/ https://semmle.com/)? If so, how does Semgrep compare to CodeQL? Edit: There is a help entry: https://semgrep.dev/docs/faq/#how-is-semgrep-different-from-codeql https://semgrep.dev/docs/faq/#how-is-semgrep-different-from-...

by ievans 6y ago

carlmr 6y ago

First of all, I love the idea of semgrep, but can't use it since we're using C++. Is there any chance for C++ support in the future?

lvh 6y ago
Not GP, but one the one hand semgrep has a real honest to goodness parser at its core; on the other hand I'd expect C++ to have sufficiently complicated semantics that it needs some understanding of C++ specific mechanics to be useful. Furthermore, you'd need preprocessor and template expansion magic to really get to the bottom of it. Effectively this is the same problem e.g. javacpp has.
ievans 6y ago
The good news is that we’ve replaced almost all the homegrown parsers that were written while the tool was at Facebook and we’re using the now tree-sitter project, which already has parsers for 40+ languages. There is a tree-sitter-cpp project we can and will eventually integrate! The bad news is this requires the code to not use heavily macros to be parseable as-is. So really the difficulty is not C++ but rather the pre-processor.
- carlmr 6y ago
  That sounds really cool, would it be possible to ignore the macros or can the code not be parsed at all then?
- philsnow 6y ago
  I guess you could run the preprocessor and then run tree-sitter-cpp / semgrep on the preprocessed output, but the problem would then be trying to tie any findings from that to the original source. Do gcc/clang/any other preprocessor create "source maps" that could facilitate that? GCC looks like it has a `-fdebug-cpp` that "[...] dumps debugging information about location maps. Every token in the output is preceded by the dump of the map its location belongs to."
  - carlmr 6y ago
    The preprocessed output of GCC and Clang usually contains the file names.
    - inetknght 6y ago
      Better than that, compilers can be told to include line numbers too. This thread would love to learn about Compiler Explorer https://gcc.godbolt.org/ https://gcc.godbolt.org/ which works for C++ and many other languages.
      - philsnow 6y ago
        Line numbers are a good start, but JS source maps go from output source byte ranges to input source byte ranges. I don't write or even read a lot of C++ these days but I recall from when I did that a major pain point was deciphering compiler warnings/errors when there are a lot of templates, macros, or both. Seems like the problem has been around forever.
  - lvh 6y ago
    Right; but this turns out to be pretty tricky in practice. I've attempted to do this for even relatively straightforward code (libsodium--complex in implementation, though not in API) with libclang and it was not particularly pleasant. Some prior art for reference: https://github.com/bytedeco/javacpp/issues/51 https://github.com/bytedeco/javacpp/issues/51

dti 6y ago

Is Semmle, offering CodeQL language and LGTM service, and recently acquired by Github, doing a similar thing (https://semmle.com/ https://semmle.com/)? If so, how does Semgrep compare to CodeQL? Edit: There is a help entry: https://semgrep.dev/docs/faq/#how-is-semgrep-different-from-codeql https://semgrep.dev/docs/faq/#how-is-semgrep-different-from-...