7 ms·

One of the nice things about Bazel that the article didn't get a chance to go into is it uses SHA hashes of files, rather than file timestamps, to determine whe

by jart 4y ago

One of the nice things about Bazel that the article didn't get a chance to go into is it uses SHA hashes of files, rather than file timestamps, to determine when an artifact has changed and therefore needs to be updated. It's slightly more costly to compute hashes, but it's necessary if you want to have something like a global cache of build artifacts, since synchronizing time across machines is hard. What I'd recommend for anyone really, is to just do what Google did. For the first six years of Google's lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. I'm obviously not there since I'm just a scrappy open source coder. So for me I'm quite happy to be working with GNU Make and I can foresee myself getting many additional years of use out of it.

gravypod 4y ago
> What I'd recommend for anyone really, is to just do what Google did. For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. Hopefully as a community we can build things that are scalable and as-simple-as Make. I think please.build is a step in the right direction but still too complicated.
- chubot 4y ago
  Eh Make is not good for most problems -- see experiences here by dwheeler, me, frankohn: https://news.ycombinator.com/item?id=32301606 https://news.ycombinator.com/item?id=32301606 And this other thread I linked there, with feedback from FreeBSD engineers: https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc --- Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make
  - deleted 4y ago
    [deleted]
  - tux1968 4y ago
    > Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Are you suggesting it's easy to learn how to hand code Ninja files, or do you have a different easy-to-learn tool to generate them, in mind?
    - chubot 4y ago
      Yeah it's true, on the face of it, they aren't directly comparable. You do need a generator and GNU Make has more features. There's some more color here: https://news.ycombinator.com/item?id=32382760 https://news.ycombinator.com/item?id=32382760 (I borrowed ninja_syntax.py, and it worked very well, but I don't claim that will work for everybody.) However I also cringe when people say to write GNU makefiles from scratch -- because those ALSO don't work for everybody, and are generated more often than not (e.g. by autotools, CMake, kconfig, etc.) So I'd say that GNU make gives you the illusion that you can just use that one tool, but you often end up needing to generate it anyway. And then you should have used Ninja with whatever generator you ended up with :)
  - dwheeler 4y ago
    GNU make is great for many things, used correctly. The problem is that POSIX make is extremely impoverished, so sticking to just the POSIX subset is often a bad idea. In many cases using "make" should really mean using "GNU make" so you can use conditionals, automated dependendency generation (via reloads of dependency info), etc.
  - gravypod 4y ago
    (Opinions are my own) > Eh Make is not good for most problems Agreed. I spend a lot of time advocating for Bazel for this reason. > Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Ninja seems more like a side-grade than a wholesale improvement. It does not fix all the problems that need to be addressed. Compare the two: * https://ninja-build.org/manual.html#_philosophical_overview https://ninja-build.org/manual.html#_philosophical_overview * https://bazel.build/start/bazel-intro https://bazel.build/start/bazel-intro Right now Bazel's biggest issues are: 1. Getting started is difficult: There is a lot of impedance mismatch across various language toolchains. This can be fixed over time by improving the tooling/libraries/docs. If JetBrains had a "New Bazel Project" and had a way to 1-click-add Java/Python/Ruby/golang/Rust/etc source into //third_party with BUILD files and a version of build_cleaner Bazel would win. Just making it easy to vendor libraries and build software from the IDE is I think all it would take to get popular. 2. The JVM: I am a huge fan of Java (I've build multiple company's backends in it) but it is not a great choice for CLI tooling (even with a daemon). A no-dependency, small, build system executable would go a long way to making it easy to get people started. 3. The cruft: A lot of things in Bazel are the way they are because someone had to get a feature out ASAP to support $BIG_USER to do $BIG_THING and it would be too difficult to migrate away/to something at Google. If we drop the cruft and redesign things from the ground up we can get a nice abstraction for the world. For example, please.build's proto_library is VERY easy to use (way easier than even the Google internal versions imo). 4. The tooling: You can get massive CI improvements, free mutation testing, frameworks for build (integration tests, e2e tests, etc), and much more by building things against the Bazel API. Unfortunately not much outside of Google supports this API. Example of things you can do with this API: https://kythe.io/examples/#extracting-compilations-using-bazel https://kythe.io/examples/#extracting-compilations-using-baz... We could live in a world where people can build langauge-agnostic tooling that automatically works so long as you pass it a `*_binary` target that transparently builds for all major platforms (zig's or APE's as crosstool) which would allow platform vendors to define macros to completely automate deployment and development for their systems. For example we could have: ``` from aws import lambda lambda.package( name = "my_api", deps = [":api_handler"], ) ``` And just by running `build :my_api` you could have a zip file which packages your code (from any language) into a lambda.
  - thayne 4y ago
    Ninja is self described as an assembler for builds. You aren't expected to write ninja build files manually.
jeffreygoesto 4y ago
SCons [0] uses something similar (MD5 instead of SHA) since the end of the 90es, so at least that aspect is not a Google invention. The cache there is local though. We never had flaky builds with it and could extend it very nicely, unfortunately it was not very fast. [0] https://en.m.wikipedia.org/wiki/SCons https://en.m.wikipedia.org/wiki/SCons
chubot 4y ago
For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make ??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004. So at most there were 6 years of using Make only, i.e. "google2". Importantly, pre-Blaze google3 wasn't just GNU make -- Python was a huge part of it, which is why the Bazel build language Starlark looks like Python. It used to literally be Python, and now it's a Python-like language with parallel evaluation. --- If you want to do what "scrappy Google" did these days, then you should use Python + Ninja. Ninja is meant to be generated, just like GNU Make was generated by Python. (A big difference is that GNU make has a big database of built-in rules that basically do nothing but slow down incremental rebuilds.) I described that strategy a bit a few days ago: https://news.ycombinator.com/item?id=32307188 https://news.ycombinator.com/item?id=32307188 --- This work with Landlock looks very cool, and it would make a lot of sense for Ninja to have optional support for it. Some of the caveats are a bit scary but hopefully that can be worked out over time. The way I was thinking of doing it was just to have a ./NINJA_config.py --slow-sandbox mode. So you can use any sandbox to warn you about missing dependencies, including something container-based like bubblewrap, symlink farms, or Landlock. I think that would work, though I haven't tried it. The shared library issue is tricky, etc. It's very useful to have the build config / generator split for this reason, and many others (e.g. build variants go only in the first stage, not the second). I wrote 3 substantial GNU makefiles from scratch and regretted it largely because it lacks this split -- it has a very tortured way of doing build "metaprogramming". IIRC one dimension of variants was OK, but 2 got you into the "write a Lisp in Make" territory. Might as well use Python (or Lua, etc.)
- ErikCorry 4y ago
  I like the speed of Ninja, and the flexibility of having a stage that generates the ninja (or make) files. But it's a bit unclear to me what the best practice is for when to generate the ninja files. It feels like the programmer is expected to do this stage manually and there's no great automatic way to generate the ninja files when they need to be generated. How do good build systems solve this?
zvr 4y ago
(GNU) Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes. IIRC, the GNU Make Book has the code ready for you to study... Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.
mst 4y ago
... now I want landlock redo

by jart 4y ago

gravypod 4y ago

> What I'd recommend for anyone really, is to just do what Google did. For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. Hopefully as a community we can build things that are scalable and as-simple-as Make. I think please.build is a step in the right direction but still too complicated.

chubot 4y ago
Eh Make is not good for most problems -- see experiences here by dwheeler, me, frankohn: https://news.ycombinator.com/item?id=32301606 https://news.ycombinator.com/item?id=32301606 And this other thread I linked there, with feedback from FreeBSD engineers: https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc --- Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make
- deleted 4y ago
  [deleted]
- tux1968 4y ago
  > Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Are you suggesting it's easy to learn how to hand code Ninja files, or do you have a different easy-to-learn tool to generate them, in mind?
  - chubot 4y ago
    Yeah it's true, on the face of it, they aren't directly comparable. You do need a generator and GNU Make has more features. There's some more color here: https://news.ycombinator.com/item?id=32382760 https://news.ycombinator.com/item?id=32382760 (I borrowed ninja_syntax.py, and it worked very well, but I don't claim that will work for everybody.) However I also cringe when people say to write GNU makefiles from scratch -- because those ALSO don't work for everybody, and are generated more often than not (e.g. by autotools, CMake, kconfig, etc.) So I'd say that GNU make gives you the illusion that you can just use that one tool, but you often end up needing to generate it anyway. And then you should have used Ninja with whatever generator you ended up with :)
- dwheeler 4y ago
  GNU make is great for many things, used correctly. The problem is that POSIX make is extremely impoverished, so sticking to just the POSIX subset is often a bad idea. In many cases using "make" should really mean using "GNU make" so you can use conditionals, automated dependendency generation (via reloads of dependency info), etc.
- gravypod 4y ago
  (Opinions are my own) > Eh Make is not good for most problems Agreed. I spend a lot of time advocating for Bazel for this reason. > Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Ninja seems more like a side-grade than a wholesale improvement. It does not fix all the problems that need to be addressed. Compare the two: * https://ninja-build.org/manual.html#_philosophical_overview https://ninja-build.org/manual.html#_philosophical_overview * https://bazel.build/start/bazel-intro https://bazel.build/start/bazel-intro Right now Bazel's biggest issues are: 1. Getting started is difficult: There is a lot of impedance mismatch across various language toolchains. This can be fixed over time by improving the tooling/libraries/docs. If JetBrains had a "New Bazel Project" and had a way to 1-click-add Java/Python/Ruby/golang/Rust/etc source into //third_party with BUILD files and a version of build_cleaner Bazel would win. Just making it easy to vendor libraries and build software from the IDE is I think all it would take to get popular. 2. The JVM: I am a huge fan of Java (I've build multiple company's backends in it) but it is not a great choice for CLI tooling (even with a daemon). A no-dependency, small, build system executable would go a long way to making it easy to get people started. 3. The cruft: A lot of things in Bazel are the way they are because someone had to get a feature out ASAP to support $BIG_USER to do $BIG_THING and it would be too difficult to migrate away/to something at Google. If we drop the cruft and redesign things from the ground up we can get a nice abstraction for the world. For example, please.build's proto_library is VERY easy to use (way easier than even the Google internal versions imo). 4. The tooling: You can get massive CI improvements, free mutation testing, frameworks for build (integration tests, e2e tests, etc), and much more by building things against the Bazel API. Unfortunately not much outside of Google supports this API. Example of things you can do with this API: https://kythe.io/examples/#extracting-compilations-using-bazel https://kythe.io/examples/#extracting-compilations-using-baz... We could live in a world where people can build langauge-agnostic tooling that automatically works so long as you pass it a `*_binary` target that transparently builds for all major platforms (zig's or APE's as crosstool) which would allow platform vendors to define macros to completely automate deployment and development for their systems. For example we could have: ``` from aws import lambda lambda.package( name = "my_api", deps = [":api_handler"], ) ``` And just by running `build :my_api` you could have a zip file which packages your code (from any language) into a lambda.
- thayne 4y ago
  Ninja is self described as an assembler for builds. You aren't expected to write ninja build files manually.

jeffreygoesto 4y ago

SCons [0] uses something similar (MD5 instead of SHA) since the end of the 90es, so at least that aspect is not a Google invention. The cache there is local though. We never had flaky builds with it and could extend it very nicely, unfortunately it was not very fast. [0] https://en.m.wikipedia.org/wiki/SCons https://en.m.wikipedia.org/wiki/SCons

chubot 4y ago

For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make ??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004. So at most there were 6 years of using Make only, i.e. "google2". Importantly, pre-Blaze google3 wasn't just GNU make -- Python was a huge part of it, which is why the Bazel build language Starlark looks like Python. It used to literally be Python, and now it's a Python-like language with parallel evaluation. --- If you want to do what "scrappy Google" did these days, then you should use Python + Ninja. Ninja is meant to be generated, just like GNU Make was generated by Python. (A big difference is that GNU make has a big database of built-in rules that basically do nothing but slow down incremental rebuilds.) I described that strategy a bit a few days ago: https://news.ycombinator.com/item?id=32307188 https://news.ycombinator.com/item?id=32307188 --- This work with Landlock looks very cool, and it would make a lot of sense for Ninja to have optional support for it. Some of the caveats are a bit scary but hopefully that can be worked out over time. The way I was thinking of doing it was just to have a ./NINJA_config.py --slow-sandbox mode. So you can use any sandbox to warn you about missing dependencies, including something container-based like bubblewrap, symlink farms, or Landlock. I think that would work, though I haven't tried it. The shared library issue is tricky, etc. It's very useful to have the build config / generator split for this reason, and many others (e.g. build variants go only in the first stage, not the second). I wrote 3 substantial GNU makefiles from scratch and regretted it largely because it lacks this split -- it has a very tortured way of doing build "metaprogramming". IIRC one dimension of variants was OK, but 2 got you into the "write a Lisp in Make" territory. Might as well use Python (or Lua, etc.)

ErikCorry 4y ago
I like the speed of Ninja, and the flexibility of having a stage that generates the ninja (or make) files. But it's a bit unclear to me what the best practice is for when to generate the ninja files. It feels like the programmer is expected to do this stage manually and there's no great automatic way to generate the ninja files when they need to be generated. How do good build systems solve this?

zvr 4y ago

(GNU) Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes. IIRC, the GNU Make Book has the code ready for you to study... Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.

mst 4y ago

... now I want landlock redo