7 ms·

This is definitely interesting and cool, however: > Landlock Make can build code five times faster than Bazel, while offering the same advantages in terms of s

This is definitely interesting and cool, however: > Landlock Make can build code five times faster than Bazel, while offering the same advantages in terms of safety. In other words, you get all the benefits of a big corporation build system, in a tiny lightweight binary that any indie developer can love. In terms of safety, maybe almost (bazel can check if the source files changed during the build, this (afaict) can not). But bazel also provides a lot more (caching, remote builds, ...). So, while cool, read more on it and evaluate it in depth before deciding to replace bazel with this.

jart 4y ago
One of the nice things about Bazel that the article didn't get a chance to go into is it uses SHA hashes of files, rather than file timestamps, to determine when an artifact has changed and therefore needs to be updated. It's slightly more costly to compute hashes, but it's necessary if you want to have something like a global cache of build artifacts, since synchronizing time across machines is hard. What I'd recommend for anyone really, is to just do what Google did. For the first six years of Google's lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. I'm obviously not there since I'm just a scrappy open source coder. So for me I'm quite happy to be working with GNU Make and I can foresee myself getting many additional years of use out of it.
- gravypod 4y ago
  > What I'd recommend for anyone really, is to just do what Google did. For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. Hopefully as a community we can build things that are scalable and as-simple-as Make. I think please.build is a step in the right direction but still too complicated.
  - chubot 4y ago
    Eh Make is not good for most problems -- see experiences here by dwheeler, me, frankohn: https://news.ycombinator.com/item?id=32301606 https://news.ycombinator.com/item?id=32301606 And this other thread I linked there, with feedback from FreeBSD engineers: https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc --- Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make
    - deleted 4y ago
      [deleted]
    - tux1968 4y ago
      > Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Are you suggesting it's easy to learn how to hand code Ninja files, or do you have a different easy-to-learn tool to generate them, in mind?
      - chubot 4y ago
        Yeah it's true, on the face of it, they aren't directly comparable. You do need a generator and GNU Make has more features. There's some more color here: https://news.ycombinator.com/item?id=32382760 https://news.ycombinator.com/item?id=32382760 (I borrowed ninja_syntax.py, and it worked very well, but I don't claim that will work for everybody.) However I also cringe when people say to write GNU makefiles from scratch -- because those ALSO don't work for everybody, and are generated more often than not (e.g. by autotools, CMake, kconfig, etc.) So I'd say that GNU make gives you the illusion that you can just use that one tool, but you often end up needing to generate it anyway. And then you should have used Ninja with whatever generator you ended up with :)
    - dwheeler 4y ago
      GNU make is great for many things, used correctly. The problem is that POSIX make is extremely impoverished, so sticking to just the POSIX subset is often a bad idea. In many cases using "make" should really mean using "GNU make" so you can use conditionals, automated dependendency generation (via reloads of dependency info), etc.
- jeffreygoesto 4y ago
  SCons [0] uses something similar (MD5 instead of SHA) since the end of the 90es, so at least that aspect is not a Google invention. The cache there is local though. We never had flaky builds with it and could extend it very nicely, unfortunately it was not very fast. [0] https://en.m.wikipedia.org/wiki/SCons https://en.m.wikipedia.org/wiki/SCons
- chubot 4y ago
  For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make ??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004. So at most there were 6 years of using Make only, i.e. "google2". Importantly, pre-Blaze google3 wasn't just GNU make -- Python was a huge part of it, which is why the Bazel build language Starlark looks like Python. It used to literally be Python, and now it's a Python-like language with parallel evaluation. --- If you want to do what "scrappy Google" did these days, then you should use Python + Ninja. Ninja is meant to be generated, just like GNU Make was generated by Python. (A big difference is that GNU make has a big database of built-in rules that basically do nothing but slow down incremental rebuilds.) I described that strategy a bit a few days ago: https://news.ycombinator.com/item?id=32307188 https://news.ycombinator.com/item?id=32307188 --- This work with Landlock looks very cool, and it would make a lot of sense for Ninja to have optional support for it. Some of the caveats are a bit scary but hopefully that can be worked out over time. The way I was thinking of doing it was just to have a ./NINJA_config.py --slow-sandbox mode. So you can use any sandbox to warn you about missing dependencies, including something container-based like bubblewrap, symlink farms, or Landlock. I think that would work, though I haven't tried it. The shared library issue is tricky, etc. It's very useful to have the build config / generator split for this reason, and many others (e.g. build variants go only in the first stage, not the second). I wrote 3 substantial GNU makefiles from scratch and regretted it largely because it lacks this split -- it has a very tortured way of doing build "metaprogramming". IIRC one dimension of variants was OK, but 2 got you into the "write a Lisp in Make" territory. Might as well use Python (or Lua, etc.)
  - ErikCorry 4y ago
    I like the speed of Ninja, and the flexibility of having a stage that generates the ninja (or make) files. But it's a bit unclear to me what the best practice is for when to generate the ninja files. It feels like the programmer is expected to do this stage manually and there's no great automatic way to generate the ninja files when they need to be generated. How do good build systems solve this?
- zvr 4y ago
  (GNU) Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes. IIRC, the GNU Make Book has the code ready for you to study... Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.
- mst 4y ago
  ... now I want landlock redo

jart 4y ago

One of the nice things about Bazel that the article didn't get a chance to go into is it uses SHA hashes of files, rather than file timestamps, to determine when an artifact has changed and therefore needs to be updated. It's slightly more costly to compute hashes, but it's necessary if you want to have something like a global cache of build artifacts, since synchronizing time across machines is hard. What I'd recommend for anyone really, is to just do what Google did. For the first six years of Google's lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. I'm obviously not there since I'm just a scrappy open source coder. So for me I'm quite happy to be working with GNU Make and I can foresee myself getting many additional years of use out of it.

gravypod 4y ago
> What I'd recommend for anyone really, is to just do what Google did. For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. Hopefully as a community we can build things that are scalable and as-simple-as Make. I think please.build is a step in the right direction but still too complicated.
- chubot 4y ago
  Eh Make is not good for most problems -- see experiences here by dwheeler, me, frankohn: https://news.ycombinator.com/item?id=32301606 https://news.ycombinator.com/item?id=32301606 And this other thread I linked there, with feedback from FreeBSD engineers: https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc https://lobste.rs/s/7svvkz/using_bsd_make#c_bfwcyc --- Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make
  - deleted 4y ago
    [deleted]
  - tux1968 4y ago
    > Ninja basically gives you the parts of GNU make that are good, without the cruft and slowness. And you can learn it in 20 minutes, unlike GNU make Are you suggesting it's easy to learn how to hand code Ninja files, or do you have a different easy-to-learn tool to generate them, in mind?
    - chubot 4y ago
      Yeah it's true, on the face of it, they aren't directly comparable. You do need a generator and GNU Make has more features. There's some more color here: https://news.ycombinator.com/item?id=32382760 https://news.ycombinator.com/item?id=32382760 (I borrowed ninja_syntax.py, and it worked very well, but I don't claim that will work for everybody.) However I also cringe when people say to write GNU makefiles from scratch -- because those ALSO don't work for everybody, and are generated more often than not (e.g. by autotools, CMake, kconfig, etc.) So I'd say that GNU make gives you the illusion that you can just use that one tool, but you often end up needing to generate it anyway. And then you should have used Ninja with whatever generator you ended up with :)
  - dwheeler 4y ago
    GNU make is great for many things, used correctly. The problem is that POSIX make is extremely impoverished, so sticking to just the POSIX subset is often a bad idea. In many cases using "make" should really mean using "GNU make" so you can use conditionals, automated dependendency generation (via reloads of dependency info), etc.
jeffreygoesto 4y ago
SCons [0] uses something similar (MD5 instead of SHA) since the end of the 90es, so at least that aspect is not a Google invention. The cache there is local though. We never had flaky builds with it and could extend it very nicely, unfortunately it was not very fast. [0] https://en.m.wikipedia.org/wiki/SCons https://en.m.wikipedia.org/wiki/SCons
chubot 4y ago
For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make ??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004. So at most there were 6 years of using Make only, i.e. "google2". Importantly, pre-Blaze google3 wasn't just GNU make -- Python was a huge part of it, which is why the Bazel build language Starlark looks like Python. It used to literally be Python, and now it's a Python-like language with parallel evaluation. --- If you want to do what "scrappy Google" did these days, then you should use Python + Ninja. Ninja is meant to be generated, just like GNU Make was generated by Python. (A big difference is that GNU make has a big database of built-in rules that basically do nothing but slow down incremental rebuilds.) I described that strategy a bit a few days ago: https://news.ycombinator.com/item?id=32307188 https://news.ycombinator.com/item?id=32307188 --- This work with Landlock looks very cool, and it would make a lot of sense for Ninja to have optional support for it. Some of the caveats are a bit scary but hopefully that can be worked out over time. The way I was thinking of doing it was just to have a ./NINJA_config.py --slow-sandbox mode. So you can use any sandbox to warn you about missing dependencies, including something container-based like bubblewrap, symlink farms, or Landlock. I think that would work, though I haven't tried it. The shared library issue is tricky, etc. It's very useful to have the build config / generator split for this reason, and many others (e.g. build variants go only in the first stage, not the second). I wrote 3 substantial GNU makefiles from scratch and regretted it largely because it lacks this split -- it has a very tortured way of doing build "metaprogramming". IIRC one dimension of variants was OK, but 2 got you into the "write a Lisp in Make" territory. Might as well use Python (or Lua, etc.)
- ErikCorry 4y ago
  I like the speed of Ninja, and the flexibility of having a stage that generates the ninja (or make) files. But it's a bit unclear to me what the best practice is for when to generate the ninja files. It feels like the programmer is expected to do this stage manually and there's no great automatic way to generate the ninja files when they need to be generated. How do good build systems solve this?
zvr 4y ago
(GNU) Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes. IIRC, the GNU Make Book has the code ready for you to study... Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.
mst 4y ago
... now I want landlock redo