6 ms·

I wish more people were taking a look at Bazel, and the Bazel-family, of build systems. I've looked at Meson/Ninja/Make and it works but it's truely very compli

I wish more people were taking a look at Bazel, and the Bazel-family, of build systems. I've looked at Meson/Ninja/Make and it works but it's truely very complicated and very difficult to get hermetic & reproducible builds from these systems. In my opinion, Bazel will be the future once the currently ongoing external dependency management design doc is implemented. I've migrated quite a few builds to bazel at work and I've seen a dramatic reduction in compile times - not even from compiling source code! Doing things like packaging zips, generating code, generating docs (swagger), etc all adds up in CI time and automating, and caching, absolutely all of it is astonishingly useful.

dcolkitt 6y ago
Out of curiosity, how difficult did you find the migration process? Especially in terms of having to restructure the project. I have no direct experience, but from what I've heard Bazel is very opinionated about structure. Which makes it particularly challenging to migrate pre-existing projects that may not conform to its view. But again this all hearsay, so I'd like to hear from someone with direct experience.
- gravypod 6y ago
  It was pretty easy for the software I was migrating because it was already in a monorepo and we were already using some very similar conventions. Anyone writing Java code that uses Gradle or Maven shouldn't have much pain migrating. I'm acutally currently supporting both Gradle and Bazel on the same backend code base to give another team (who depends on my code) a chance to gracefully migrate. The "hardest" thing to migrate has been C and Python code. For these the package for something is based on an absolute file path from the root of your repo. So `company/foobar/thing.py` is `from company.foobar import thing`. I find this very useful personally because it's now standardized through the codebase (no one just plops a setup.py somewhere and does whatever they want) but it makes the migration difficult because you need to change the paths. There's some work being done in rules_python and in cc_library/cc_binary that lets you control this behavior and hack it to align with your current company standards though which is really nice to use for a gradual migration.
  - liuliu 6y ago
    For cc_library, I think now you can add include paths so `#include "somethingUnderYourPath"` will be easier.
slezyr 6y ago
For same no one looks at the GYP, GN, and now Bazel. Google bury their build tools just as fast as other products. And something about Java dependency.
- barumi 6y ago
  To be fair, gyp is quite awful. I mean, a low-level language to generate project files for a few IDEs from a definition file where IDE-specific settings had to be added anyway? While the source of truth is left out of the equation, rendering those project files read-only and forcing developers to reconfigure their IDEs each and every single time they updated the project? This concept seems to have been thought out by masochists.
niccl 6y ago
Do you know whether/how it can deal with database tables dependent on source .csv files? My job involves lots of ETL-type work, where the extracted source file(s) for a particular table might have names that include the extract date or version, and a table might depend on other tables. I've been looking at make for dealing with the process, but automatic dependency management seems hard, and dealing with the variable source file names even harder (possibly because I'm not more than a make novice).
- brown9-2 6y ago
  Bazel likely won’t deal with this at all, but you could write your own rules to implement something like “insert to a database”.
Ericson2314 6y ago
Bazel is a decent idea but poor execution. I think Nix is the future because it already has the "external dependency" part on lock and it's just a matter of making more fine-grained stuff. Meson + Nix skipping the Ninja part I would really like to see. I prefer it when the "big dumb builder" and DSLs are developed a bit separately, too. Agreed Ninja's lack of purity is a real bummer.
- gravypod 6y ago
  I'm not so sure about Nix since it seems much more complex in presentation. The meat and potatoes of Bazel is the BUILD langauge. There may be some warts right now but the community is attempting to address these warts. Difficulty around specifying versions of packages, semantics for building containers, etc. Nix users do not seem to think it's build language is complex. I think Nix might be the worlds best attempt at something like this but it's the same reason why Haskell hasn't gotten adoption: "eeh it looks complicated" Everyone understands `some_rule(some="argument")` though.
  - barumi 6y ago
    > There may be some warts right now but the community is attempting to address these warts. Some of Bazel's design problems do feel like the outcome of a poorly executed requirements gathering process. Last time I checked, it assumed for some reason that C/C++ interface headers shared a common root folder with libraries, and if that root folder is not the immediate parent folder then it ends up including all files within that tree branch in its scope. I also recall that the process of including system libraries that don't support Bazel is needlessly convoluted, and involved adding libraries and interface headers in separate targets just to be able to control visibility. This design oversight was specially disheartening because it made it quite clear that the use case of supporting external packages benefitted from zero attention.
    - Ericson2314 6y ago
      Monorepos do have advantages, but Google fundamentally uses them as an excuse to fail ot understand modularity and composition. That's a fatal mistake.
pantsbuild 6y ago
In the general category of Bazel-alike build systems, I'd humbly submit that Pants v2 is worth taking a look at. It's a generic, high performance build system for large projects, implemented in Rust with a Python plugin API. The project had a large announcement today in fact! https://news.ycombinator.com/item?id=24911148 https://news.ycombinator.com/item?id=24911148 For supported languages (the 2.0.0 release and API is brand new, so only Python and Bash are supported so far), the amount of per-file boilerplate is nearly as low as it goes: https://www.pantsbuild.org/docs/how-does-pants-work#dependency-inference https://www.pantsbuild.org/docs/how-does-pants-work#dependen...
myzie 6y ago
Zim was inspired by all of these and has drastically cut build times for our team. We use it to build hundreds of Go, Python, and Node artifacts for use in AWS. https://github.com/fugue/zim https://github.com/fugue/zim
dastbe 6y ago
do you have a link to the design doc? EDIT: for those searching, I suspect its https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3Zu...

dcolkitt 6y ago

Out of curiosity, how difficult did you find the migration process? Especially in terms of having to restructure the project. I have no direct experience, but from what I've heard Bazel is very opinionated about structure. Which makes it particularly challenging to migrate pre-existing projects that may not conform to its view. But again this all hearsay, so I'd like to hear from someone with direct experience.

gravypod 6y ago
It was pretty easy for the software I was migrating because it was already in a monorepo and we were already using some very similar conventions. Anyone writing Java code that uses Gradle or Maven shouldn't have much pain migrating. I'm acutally currently supporting both Gradle and Bazel on the same backend code base to give another team (who depends on my code) a chance to gracefully migrate. The "hardest" thing to migrate has been C and Python code. For these the package for something is based on an absolute file path from the root of your repo. So `company/foobar/thing.py` is `from company.foobar import thing`. I find this very useful personally because it's now standardized through the codebase (no one just plops a setup.py somewhere and does whatever they want) but it makes the migration difficult because you need to change the paths. There's some work being done in rules_python and in cc_library/cc_binary that lets you control this behavior and hack it to align with your current company standards though which is really nice to use for a gradual migration.
- liuliu 6y ago
  For cc_library, I think now you can add include paths so `#include "somethingUnderYourPath"` will be easier.

slezyr 6y ago

For same no one looks at the GYP, GN, and now Bazel. Google bury their build tools just as fast as other products. And something about Java dependency.

barumi 6y ago
To be fair, gyp is quite awful. I mean, a low-level language to generate project files for a few IDEs from a definition file where IDE-specific settings had to be added anyway? While the source of truth is left out of the equation, rendering those project files read-only and forcing developers to reconfigure their IDEs each and every single time they updated the project? This concept seems to have been thought out by masochists.

niccl 6y ago

Do you know whether/how it can deal with database tables dependent on source .csv files? My job involves lots of ETL-type work, where the extracted source file(s) for a particular table might have names that include the extract date or version, and a table might depend on other tables. I've been looking at make for dealing with the process, but automatic dependency management seems hard, and dealing with the variable source file names even harder (possibly because I'm not more than a make novice).

brown9-2 6y ago
Bazel likely won’t deal with this at all, but you could write your own rules to implement something like “insert to a database”.

Ericson2314 6y ago

Bazel is a decent idea but poor execution. I think Nix is the future because it already has the "external dependency" part on lock and it's just a matter of making more fine-grained stuff. Meson + Nix skipping the Ninja part I would really like to see. I prefer it when the "big dumb builder" and DSLs are developed a bit separately, too. Agreed Ninja's lack of purity is a real bummer.

gravypod 6y ago
I'm not so sure about Nix since it seems much more complex in presentation. The meat and potatoes of Bazel is the BUILD langauge. There may be some warts right now but the community is attempting to address these warts. Difficulty around specifying versions of packages, semantics for building containers, etc. Nix users do not seem to think it's build language is complex. I think Nix might be the worlds best attempt at something like this but it's the same reason why Haskell hasn't gotten adoption: "eeh it looks complicated" Everyone understands `some_rule(some="argument")` though.
- barumi 6y ago
  > There may be some warts right now but the community is attempting to address these warts. Some of Bazel's design problems do feel like the outcome of a poorly executed requirements gathering process. Last time I checked, it assumed for some reason that C/C++ interface headers shared a common root folder with libraries, and if that root folder is not the immediate parent folder then it ends up including all files within that tree branch in its scope. I also recall that the process of including system libraries that don't support Bazel is needlessly convoluted, and involved adding libraries and interface headers in separate targets just to be able to control visibility. This design oversight was specially disheartening because it made it quite clear that the use case of supporting external packages benefitted from zero attention.
  - Ericson2314 6y ago
    Monorepos do have advantages, but Google fundamentally uses them as an excuse to fail ot understand modularity and composition. That's a fatal mistake.

pantsbuild 6y ago

In the general category of Bazel-alike build systems, I'd humbly submit that Pants v2 is worth taking a look at. It's a generic, high performance build system for large projects, implemented in Rust with a Python plugin API. The project had a large announcement today in fact! https://news.ycombinator.com/item?id=24911148 https://news.ycombinator.com/item?id=24911148 For supported languages (the 2.0.0 release and API is brand new, so only Python and Bash are supported so far), the amount of per-file boilerplate is nearly as low as it goes: https://www.pantsbuild.org/docs/how-does-pants-work#dependency-inference https://www.pantsbuild.org/docs/how-does-pants-work#dependen...

myzie 6y ago

Zim was inspired by all of these and has drastically cut build times for our team. We use it to build hundreds of Go, Python, and Node artifacts for use in AWS. https://github.com/fugue/zim https://github.com/fugue/zim

dastbe 6y ago

do you have a link to the design doc? EDIT: for those searching, I suspect its https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3Zu...