5 ms·

Bear is a tool that generates a compilation database for Clang tooling

47 points by nateb2022 2y ago

synergy20 2y ago
the compilation database is essential for editors these days, cmake can generate it directly but not all code use cmake, in that case I just use compiledb, which is inactive (there is a newer compiledb-go though), then use bear. somehow bear did not work well for me comparing to compiledb so compiledb(now compiledb-go) is my main tool now.
AceJohnny2 2y ago
If you have control over the build system, clang can generate that directly nowadays, by adding -MJ path to CFLAGS (or equiv). This will save JSON fragments to path (make'em separate for each compiled file), then you can concatenate all the fragments into a compilation database. I use this approach because on macOS, with SIP, Bear no longer works as it is forbidden from injecting its hooks into the system-protected clang shim. There may be other solutions. (I haven't explored the space in a few years. They've historically fared poorly with embedded targets where one file may be compiled many different ways for targeting different platforms)
- maccard 2y ago
  One of the major problems approach with this is that you need to run the compile before you can generate this database.
  - mathstuf 2y ago
    You need to, in general, run the build for compilation databases to actually work anyways. Consider the following cases: - headers generated during the build: tools may find the wrong header or fail when the header isn't found - generated sources: does the source file in the database even exist yet? For those cases, one still needs to consider the "out of date" situation where a previous build left an old version around. This is something tools can't really even know today. Beyond that, modules present an issue. The command line doesn't have any indication of what `import foo;` means to the TU in question (and in CMake's case, the file which does tell it that information is build-time generated anyways). Even if it did, the BMI may be in an incompatible format (e.g., using GCC for the build, but using `clang-analyzer` on the database) so the tool needs to go and make its own BMI. I gave a talk at CppCon about "build databases" which is basically "compilation databases, but consider what it means to support modules". (The video isn't available yet.) I'm also working on getting a paper on them through ISO C++'s SG15 (Tooling) for proper specification. FD: CMake developer
  - deleted 2y ago
    [deleted]
  - suprjami 2y ago
    How do you propose to generate the compilation database without compiling? I would really like that for the Linux kernel.
    - xxpor 2y ago
      You really need build system support. For example, Ninja can generate the file without actually doing the build because it knows everything it _would_ do.
  - wakawaka28 2y ago
    Is this really a major problem? Is there any tool that supports generating the database without compiling? I think the answer to both of these is "no". In fact I can't even think of a reasonable case where it would be even a minor problem.
    - jchw 2y ago
      Ninja and CMake do. It's especially useful if you want a compilation database for a project that takes an extremely long time to build.
      - wakawaka28 2y ago
        CMake does but I think you have to attempt a build to get the database. On the other hand I don't think that should be necessary, as it is capable of generating Ninja build files. Almost nobody is writing Ninja files by hand. If you have to write something along those lines by hand, Makefiles would make more sense than Ninja. If Ninja does support exporting commands, it's a use case that doesn't matter because almost everyone uses CMake-generated Ninja files.
        jchw 2y ago
        You do not need to attempt a build with CMake. Also, that's kind of the point of mentioning Ninja. Inherently, anything that uses Ninja is in the same boat and doesn't need to start a build.
        wakawaka28 2y ago
        The only thing I know that uses Ninja is CMake and I'm sure CMake does not require Ninja to generate the database.
        jchw 2y ago
        The most obvious example is the build system of Chromium, which uses gn. They do have a wrapper script to actually generate the compile_commands.json file, but it's just a thin wrapper around Ninja's built-in capabilities. (I'm not sure that CMake's isn't, on that note; I'm pretty sure it's ability to build the database is in fact dependent on which generator you're using.)
- rbalint 2y ago
  If you would like to make similar hooks work again on macOS, check out this guide: https://firebuild.com/setup-macos https://firebuild.com/setup-macos . Firebuild uses similar technique to explore the process tree and create a report with CPU utilizations highlighted in a graphical process tree graph.
BoingBoomTschak 2y ago
Just shilling a single-file almost POSIX sh script I made to do the same: https://git.sr.ht/~q3cpma/scripts/tree/master/item/mkcdb https://git.sr.ht/~q3cpma/scripts/tree/master/item/mkcdb
jchw 2y ago
I frequently use this to make clangd work well. With CMake-based projects, you can just set CMAKE_EXPORT_COMPILE_COMMANDS then create a symlink from compile_commands.json in your source directory to the CMake build directory. (If you control the CMake file, you can even set this by default and make CMake create this symlink automatically.) To keep the symlink out of Git, you can add it to .gitignore of course, or if you don't control that, the lesser-known local equivalent, .git/info/exclude. I use this often in combination with direnv and Nix flakes for a good developer experience. (Note that if you need to keep the Nix flake VCS-ignored, you'll need to tell direnv to explicitly not use a Git fetcher for Nix using something like `use flake path://$PWD` though this isn't needed if you can just re-use a Nixpkgs expression, e.g. `use flake nixpkgs#wineWow64Packages.wine-unstable` or so.) One thing that sucks is that it doesn't seem to be easy to handle cross-compilation stuff. Wine is a particularly challenging case, as many of the binaries are now cross-compiled with MinGW. It still provides better completion than nothing, but I do wish I could get it to be perfect. When using Nix with MinGW I struggled even harder because Nix's MinGW is even weirder... But with enough mangling, you can even convince clangd on Linux to give somewhat decent completions for a MinGW compilation. One obvious disadvantage is that you kind of need a full rebuild to get the compilation database to work correctly. At least in my personal experience, running bear on a partial build seems to not work in an additive fashion, though maybe I was holding it wrong.
- BoingBoomTschak 2y ago
  Never thought about wanting it to work with incremental builds, so thanks for the idea. Added a jq one-line to do so in my script: https://git.sr.ht/~q3cpma/scripts/commit/3f860728eb2ea6d46e47f260bf2ca98bb2788b51 https://git.sr.ht/~q3cpma/scripts/commit/3f860728eb2ea6d46e4...
aulin 2y ago
Working with embedded I had mixed success with bear. Old environments that cannot run the latest version, LD preload messing with vendor toolchains. These days I found my peace just writing my own python scripts to parse verbose build logs. You just need to extract filename, base dir, compiler and arguments. Sometimes you're lucky and build system supports dry run and you don't even have to run a real build to get the logs. Way less invasive, no need to ask devops for additional tools and you can adapt it to any build system. Whenever I start with a new toolchain, I spend a couple of hours to tweak the parser and I'm good to go.
ktm5j 2y ago
I use bear to make a compile_comands.json file that makes the lsp work in my spacemacs setup, it's pretty cool! https://develop.spacemacs.org/layers/+lang/c-c++/README.html https://develop.spacemacs.org/layers/+lang/c-c++/README.html

47 points by nateb2022 2y ago

the compilation database is essential for editors these days, cmake can generate it directly but not all code use cmake, in that case I just use compiledb, which is inactive (there is a newer compiledb-go though), then use bear. somehow bear did not work well for me comparing to compiledb so compiledb(now compiledb-go) is my main tool now.

AceJohnny2 2y ago

If you have control over the build system, clang can generate that directly nowadays, by adding -MJ path to CFLAGS (or equiv). This will save JSON fragments to path (make'em separate for each compiled file), then you can concatenate all the fragments into a compilation database. I use this approach because on macOS, with SIP, Bear no longer works as it is forbidden from injecting its hooks into the system-protected clang shim. There may be other solutions. (I haven't explored the space in a few years. They've historically fared poorly with embedded targets where one file may be compiled many different ways for targeting different platforms)

maccard 2y ago
One of the major problems approach with this is that you need to run the compile before you can generate this database.
- mathstuf 2y ago
  You need to, in general, run the build for compilation databases to actually work anyways. Consider the following cases: - headers generated during the build: tools may find the wrong header or fail when the header isn't found - generated sources: does the source file in the database even exist yet? For those cases, one still needs to consider the "out of date" situation where a previous build left an old version around. This is something tools can't really even know today. Beyond that, modules present an issue. The command line doesn't have any indication of what `import foo;` means to the TU in question (and in CMake's case, the file which does tell it that information is build-time generated anyways). Even if it did, the BMI may be in an incompatible format (e.g., using GCC for the build, but using `clang-analyzer` on the database) so the tool needs to go and make its own BMI. I gave a talk at CppCon about "build databases" which is basically "compilation databases, but consider what it means to support modules". (The video isn't available yet.) I'm also working on getting a paper on them through ISO C++'s SG15 (Tooling) for proper specification. FD: CMake developer
- deleted 2y ago
  [deleted]
- suprjami 2y ago
  How do you propose to generate the compilation database without compiling? I would really like that for the Linux kernel.
  - xxpor 2y ago
    You really need build system support. For example, Ninja can generate the file without actually doing the build because it knows everything it _would_ do.
- wakawaka28 2y ago
  Is this really a major problem? Is there any tool that supports generating the database without compiling? I think the answer to both of these is "no". In fact I can't even think of a reasonable case where it would be even a minor problem.
  - jchw 2y ago
    Ninja and CMake do. It's especially useful if you want a compilation database for a project that takes an extremely long time to build.
    - wakawaka28 2y ago
      CMake does but I think you have to attempt a build to get the database. On the other hand I don't think that should be necessary, as it is capable of generating Ninja build files. Almost nobody is writing Ninja files by hand. If you have to write something along those lines by hand, Makefiles would make more sense than Ninja. If Ninja does support exporting commands, it's a use case that doesn't matter because almost everyone uses CMake-generated Ninja files.
      - jchw 2y ago
        You do not need to attempt a build with CMake. Also, that's kind of the point of mentioning Ninja. Inherently, anything that uses Ninja is in the same boat and doesn't need to start a build.
        wakawaka28 2y ago
        The only thing I know that uses Ninja is CMake and I'm sure CMake does not require Ninja to generate the database.
        jchw 2y ago
        The most obvious example is the build system of Chromium, which uses gn. They do have a wrapper script to actually generate the compile_commands.json file, but it's just a thin wrapper around Ninja's built-in capabilities. (I'm not sure that CMake's isn't, on that note; I'm pretty sure it's ability to build the database is in fact dependent on which generator you're using.)
rbalint 2y ago
If you would like to make similar hooks work again on macOS, check out this guide: https://firebuild.com/setup-macos https://firebuild.com/setup-macos . Firebuild uses similar technique to explore the process tree and create a report with CPU utilizations highlighted in a graphical process tree graph.

BoingBoomTschak 2y ago

Just shilling a single-file almost POSIX sh script I made to do the same: https://git.sr.ht/~q3cpma/scripts/tree/master/item/mkcdb https://git.sr.ht/~q3cpma/scripts/tree/master/item/mkcdb

jchw 2y ago

I frequently use this to make clangd work well. With CMake-based projects, you can just set CMAKE_EXPORT_COMPILE_COMMANDS then create a symlink from compile_commands.json in your source directory to the CMake build directory. (If you control the CMake file, you can even set this by default and make CMake create this symlink automatically.) To keep the symlink out of Git, you can add it to .gitignore of course, or if you don't control that, the lesser-known local equivalent, .git/info/exclude. I use this often in combination with direnv and Nix flakes for a good developer experience. (Note that if you need to keep the Nix flake VCS-ignored, you'll need to tell direnv to explicitly not use a Git fetcher for Nix using something like `use flake path://$PWD` though this isn't needed if you can just re-use a Nixpkgs expression, e.g. `use flake nixpkgs#wineWow64Packages.wine-unstable` or so.) One thing that sucks is that it doesn't seem to be easy to handle cross-compilation stuff. Wine is a particularly challenging case, as many of the binaries are now cross-compiled with MinGW. It still provides better completion than nothing, but I do wish I could get it to be perfect. When using Nix with MinGW I struggled even harder because Nix's MinGW is even weirder... But with enough mangling, you can even convince clangd on Linux to give somewhat decent completions for a MinGW compilation. One obvious disadvantage is that you kind of need a full rebuild to get the compilation database to work correctly. At least in my personal experience, running bear on a partial build seems to not work in an additive fashion, though maybe I was holding it wrong.

BoingBoomTschak 2y ago
Never thought about wanting it to work with incremental builds, so thanks for the idea. Added a jq one-line to do so in my script: https://git.sr.ht/~q3cpma/scripts/commit/3f860728eb2ea6d46e47f260bf2ca98bb2788b51 https://git.sr.ht/~q3cpma/scripts/commit/3f860728eb2ea6d46e4...

aulin 2y ago

Working with embedded I had mixed success with bear. Old environments that cannot run the latest version, LD preload messing with vendor toolchains. These days I found my peace just writing my own python scripts to parse verbose build logs. You just need to extract filename, base dir, compiler and arguments. Sometimes you're lucky and build system supports dry run and you don't even have to run a real build to get the logs. Way less invasive, no need to ask devops for additional tools and you can adapt it to any build system. Whenever I start with a new toolchain, I spend a couple of hours to tweak the parser and I'm good to go.

ktm5j 2y ago

I use bear to make a compile_comands.json file that makes the lsp work in my spacemacs setup, it's pretty cool! https://develop.spacemacs.org/layers/+lang/c-c++/README.html https://develop.spacemacs.org/layers/+lang/c-c++/README.html