4 ms·

What is the point of compile_commands.json database? I've tried to run bear on my makefile and then i've read the resulting json. It seemed to me like bash scri

by harvie 5y ago

What is the point of compile_commands.json database? I've tried to run bear on my makefile and then i've read the resulting json. It seemed to me like bash script with extra steps. Is there some added value?

touisteur 5y ago
I think the idea is to avoid doing the whole checking dependencies, and fast-build your whole system. Suppose you build a new compiler plugin/pass, you just want to force-recompile most stuff without going through all the build and autostuff. It helps building dependency graphs, so that you can run static analysis tools. Coverity IIRC does something like that to get the list of units, and build (preprocessor directives, etc) options.
- moyix 5y ago
  Yup, pretty much. If you want a static analysis tool to be able to actually parse your C code and see it like the compiler would, it needs to have all the info that the compiler did – header files, defines (including ones passed in with -D options), what the working directory was, what environment variables were in effect, etc. Otherwise there are all sorts of problems like: - Is this code even used? (Maybe it's only included for some specific configurations, or #ifdef'd out) - What does MAGIC_MACRO(x,y) do? Maybe this is only in a header file that's generated at build time, and you can't understand/analyze code that uses it without knowing its definition. - What compiler version was used? Code may be interpreted differently depending on the compiler (particularly if gcc or clang-specific extensions are used, but also just due to things like compiler bugs or idiosyncrasies) Coverity wrote a nice article about some of these issues back in 2010: https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li... > Law: You can't check code you don't see. It seems too trite to note that checking code requires first finding it... until you try to do so consistently on many large code bases. Probably the most reliable way to check a system is to grab its code during the build process; the build system knows exactly which files are included in the system and how to compile them. This seems like a simple task. Unfortunately, it's often difficult to understand what an ad hoc, homegrown build system is doing well enough to extract this information [...] > The right approach, which we have used for the past seven years, kicks off the build process and intercepts every system call it invokes. As a result, we can see everything needed for checking, including the exact executables invoked, their command lines, the directory they run in, and the version of the compiler (needed for compiler-bug workarounds). This control makes it easy to grab and precisely check all source code, to the extent of automatically changing the language dialect on a per-file basis. [...] > Law: You can't check code you can't parse. Checking code deeply requires understanding the code's semantics. The most basic requirement is that you parse it. Parsing is considered a solved problem. Unfortunately, this view is naïve, rooted in the widely believed myth that programming languages exist. > The C language does not exist; neither does Java, C++, and C#. While a language may exist as an abstract idea, and even have a pile of paper (a standard) purporting to define it, a standard is not a compiler. What language do people write code in? The character strings accepted by their compiler. Further, they equate compilation with certification. A file their compiler does not reject has been certified as "C code" no matter how blatantly illegal its contents may be to a language scholar. Fed this illegal not-C code, a tool's C front-end will reject it. This problem is the tool's problem.
OskarS 5y ago
The main use of it is for things like language servers. In order to properly show things like compiler errors in your editor, your language server (or linting engine or whatever) needs to know EXACTLY the command you used to compile. The most obvious thing is preprocessor macros, but also things like which C/C++ standard are you using, have you turned off some number of warnings, etc. compile_commands.json is a way to solve that problem in a "build system independent" way. It's just a list of exactly what commands where used for what files. As long as you generate that file correctly, you can use whatever build system you want (cmake, hand-written make file, 1000s of lines of Perl 5, a shell script from 1983, whatever) and your editor can correctly show errors, warnings, provide auto-completion, all that good stuff. An alternative (at least for clangd) is to use compile_flags.txt, where you can just specify whatever flags you want the compiler to use EDIT: if you're asking why this person in particular wants compile_commands.json files, it seems like he just wants a unified format for the compile commands of all the files in all the packages so he can use it for his analysis or whatever.

by harvie 5y ago

touisteur 5y ago

I think the idea is to avoid doing the whole checking dependencies, and fast-build your whole system. Suppose you build a new compiler plugin/pass, you just want to force-recompile most stuff without going through all the build and autostuff. It helps building dependency graphs, so that you can run static analysis tools. Coverity IIRC does something like that to get the list of units, and build (preprocessor directives, etc) options.

moyix 5y ago
Yup, pretty much. If you want a static analysis tool to be able to actually parse your C code and see it like the compiler would, it needs to have all the info that the compiler did – header files, defines (including ones passed in with -D options), what the working directory was, what environment variables were in effect, etc. Otherwise there are all sorts of problems like: - Is this code even used? (Maybe it's only included for some specific configurations, or #ifdef'd out) - What does MAGIC_MACRO(x,y) do? Maybe this is only in a header file that's generated at build time, and you can't understand/analyze code that uses it without knowing its definition. - What compiler version was used? Code may be interpreted differently depending on the compiler (particularly if gcc or clang-specific extensions are used, but also just due to things like compiler bugs or idiosyncrasies) Coverity wrote a nice article about some of these issues back in 2010: https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li... > Law: You can't check code you don't see. It seems too trite to note that checking code requires first finding it... until you try to do so consistently on many large code bases. Probably the most reliable way to check a system is to grab its code during the build process; the build system knows exactly which files are included in the system and how to compile them. This seems like a simple task. Unfortunately, it's often difficult to understand what an ad hoc, homegrown build system is doing well enough to extract this information [...] > The right approach, which we have used for the past seven years, kicks off the build process and intercepts every system call it invokes. As a result, we can see everything needed for checking, including the exact executables invoked, their command lines, the directory they run in, and the version of the compiler (needed for compiler-bug workarounds). This control makes it easy to grab and precisely check all source code, to the extent of automatically changing the language dialect on a per-file basis. [...] > Law: You can't check code you can't parse. Checking code deeply requires understanding the code's semantics. The most basic requirement is that you parse it. Parsing is considered a solved problem. Unfortunately, this view is naïve, rooted in the widely believed myth that programming languages exist. > The C language does not exist; neither does Java, C++, and C#. While a language may exist as an abstract idea, and even have a pile of paper (a standard) purporting to define it, a standard is not a compiler. What language do people write code in? The character strings accepted by their compiler. Further, they equate compilation with certification. A file their compiler does not reject has been certified as "C code" no matter how blatantly illegal its contents may be to a language scholar. Fed this illegal not-C code, a tool's C front-end will reject it. This problem is the tool's problem.

OskarS 5y ago

The main use of it is for things like language servers. In order to properly show things like compiler errors in your editor, your language server (or linting engine or whatever) needs to know EXACTLY the command you used to compile. The most obvious thing is preprocessor macros, but also things like which C/C++ standard are you using, have you turned off some number of warnings, etc. compile_commands.json is a way to solve that problem in a "build system independent" way. It's just a list of exactly what commands where used for what files. As long as you generate that file correctly, you can use whatever build system you want (cmake, hand-written make file, 1000s of lines of Perl 5, a shell script from 1983, whatever) and your editor can correctly show errors, warnings, provide auto-completion, all that good stuff. An alternative (at least for clangd) is to use compile_flags.txt, where you can just specify whatever flags you want the compiler to use EDIT: if you're asking why this person in particular wants compile_commands.json files, it seems like he just wants a unified format for the compile commands of all the files in all the packages so he can use it for his analysis or whatever.