2 ms·

The good news is that we’ve replaced almost all the homegrown parsers that were written while the tool was at Facebook and we’re using the now tree-sitter proje

by ievans 6y ago

The good news is that we’ve replaced almost all the homegrown parsers that were written while the tool was at Facebook and we’re using the now tree-sitter project, which already has parsers for 40+ languages. There is a tree-sitter-cpp project we can and will eventually integrate! The bad news is this requires the code to not use heavily macros to be parseable as-is. So really the difficulty is not C++ but rather the pre-processor.

carlmr 6y ago
That sounds really cool, would it be possible to ignore the macros or can the code not be parsed at all then?
philsnow 6y ago
I guess you could run the preprocessor and then run tree-sitter-cpp / semgrep on the preprocessed output, but the problem would then be trying to tie any findings from that to the original source. Do gcc/clang/any other preprocessor create "source maps" that could facilitate that? GCC looks like it has a `-fdebug-cpp` that "[...] dumps debugging information about location maps. Every token in the output is preceded by the dump of the map its location belongs to."
- carlmr 6y ago
  The preprocessed output of GCC and Clang usually contains the file names.
  - inetknght 6y ago
    Better than that, compilers can be told to include line numbers too. This thread would love to learn about Compiler Explorer https://gcc.godbolt.org/ https://gcc.godbolt.org/ which works for C++ and many other languages.
    - philsnow 6y ago
      Line numbers are a good start, but JS source maps go from output source byte ranges to input source byte ranges. I don't write or even read a lot of C++ these days but I recall from when I did that a major pain point was deciphering compiler warnings/errors when there are a lot of templates, macros, or both. Seems like the problem has been around forever.
- lvh 6y ago
  Right; but this turns out to be pretty tricky in practice. I've attempted to do this for even relatively straightforward code (libsodium--complex in implementation, though not in API) with libclang and it was not particularly pleasant. Some prior art for reference: https://github.com/bytedeco/javacpp/issues/51 https://github.com/bytedeco/javacpp/issues/51

by ievans 6y ago

carlmr 6y ago

That sounds really cool, would it be possible to ignore the macros or can the code not be parsed at all then?

philsnow 6y ago

I guess you could run the preprocessor and then run tree-sitter-cpp / semgrep on the preprocessed output, but the problem would then be trying to tie any findings from that to the original source. Do gcc/clang/any other preprocessor create "source maps" that could facilitate that? GCC looks like it has a `-fdebug-cpp` that "[...] dumps debugging information about location maps. Every token in the output is preceded by the dump of the map its location belongs to."

carlmr 6y ago
The preprocessed output of GCC and Clang usually contains the file names.
- inetknght 6y ago
  Better than that, compilers can be told to include line numbers too. This thread would love to learn about Compiler Explorer https://gcc.godbolt.org/ https://gcc.godbolt.org/ which works for C++ and many other languages.
  - philsnow 6y ago
    Line numbers are a good start, but JS source maps go from output source byte ranges to input source byte ranges. I don't write or even read a lot of C++ these days but I recall from when I did that a major pain point was deciphering compiler warnings/errors when there are a lot of templates, macros, or both. Seems like the problem has been around forever.
lvh 6y ago
Right; but this turns out to be pretty tricky in practice. I've attempted to do this for even relatively straightforward code (libsodium--complex in implementation, though not in API) with libclang and it was not particularly pleasant. Some prior art for reference: https://github.com/bytedeco/javacpp/issues/51 https://github.com/bytedeco/javacpp/issues/51