5 ms·

Knit: Making a Better Make

87 points by signa11 4y ago

kensai 4y ago
Nice project. Is it ready for prime time? Btw, are there any other alternatives to be compared with it and Make?
- lantry 4y ago
  I've seen a few, but the only one I can remember off the top of my head is just[0]. [0]: https://github.com/casey/just https://github.com/casey/just
  - sontek 4y ago
    I use just and it is amazing.
  - ekidd 4y ago
    Yup! Two well-established alternatives are "rake", in the Ruby community, and "just" in the Rust community. Rake is fully programmable in Ruby. Just is a bit less flexible, but it doesn't require learning Ruby, and it's quite pleasant to use. https://ruby.github.io/rake/ https://ruby.github.io/rake/
    - lloeki 4y ago
      Also, many people using rake already are often biased by how Rails uses it, but it's much more than that, and can easily be used as a make replacement, complete with freshness checks and rules. There was an article specifically avout that on the interwebs but I can't find it anymore. These ones cover some bits as well, although they're a tad heavier on using Ruby code. https://martinfowler.com/articles/rake.html https://martinfowler.com/articles/rake.html http://www.throwtheswitch.org/build/rake http://www.throwtheswitch.org/build/rake
  - masklinn 4y ago
    It should be noted that just specifically does something make was not designed for but is often conscripted into: it’s not a build tool, it’s a command runner. So when you run a command it runs the dependencies then the commands, in full, every time. I very much prefer that usually, because either the underlying tooling handles checking if all’s right or there is no such checking to do and it would be a problem. But if you expect just to build your C project and handle not touching stuff which doesn’t need to be, you’ll be disappointed.
- summarity 4y ago
  The dependency resolution is core to https://gittup.org/tup/ https://gittup.org/tup/ which had been adopted a bit at the time it came out but since faded back into obscurity.
  - NeutralForest 4y ago
    This looks cool, how have I never seen it mentioned before
- usual_science 4y ago
  There is build2 https://build2.org/ https://build2.org/
- raja_mukherji 4y ago
  I've also written my own alternative to make called Rabs (https://github.com/wrapl/rabs https://github.com/wrapl/rabs) which uses an imperative language to specify the dependency graph as well as the build instructions. It's designed for really fast incremental builds.
jjgreen 4y ago
Looks OK but adoption will be limited by the pain felt by the users of packages by people that use it. Ah, this package looks nice, I'll have a look. What's a Knitfile? Oh, that looks quite cool; oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it. Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ...
- deleted 4y ago
  [deleted]
- pxc 4y ago
  > Ah, this package looks nice, I'll have a look. What's a Knitfile? Oh, that looks quite cool; oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it. Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ... This is a useful vignette, but I wanna pick at it a bit because at least sometimes, I don't see things going this way with me. > oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it Do you really see most devs thinking this? I don't, even though I'm someone who would build it myself. > Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ... I don't understand what the problem would be here. I maintain several out-of-tree Go packages for Nix for private use and all of them are trivial because Go packages are super easy to build. Are there really many devs out there who are so hesitant to build Go projects, of packages in all languages? With C and autotools you may already have the toolchain installed, but it's generally harder to build than Go packages.
- janslow 4y ago
  I try to use Make to bootstrap project-specific tooling when missing, by getting it to download (then verify) the tools, then delegate everything else to something like knit (or just/cargo/npm). That way, there's very few prerequisites for a new dev to install (plus you have consistent build/test/run targets for all projects). If someone wants to manage tools themselves (e.g, the no prebuilt binaries example), they can do that, then skip the Makefile entirely.
- zyedidia 4y ago
aabdelhafez 4y ago
How does it compare to just? https://github.com/casey/just https://github.com/casey/just
- neerajk 4y ago
  compare to just what? what? just what? just what?
- oftenwrong 4y ago
  Knit, like Make, is designed to provide incremental builds. In other words, a build that takes into account the graph of dependencies involved. Their goal is to only run build steps that need to be run. If a build target has already been built, and its dependencies have not changed, then the build rules associated with the target can be skipped. Just, as far as I know, does not support incremental builds. Just is more intended to provide a set of project-specific scripts that are convenient to list and run. Make is sometimes used in that way with "phony" targets, but that is more of an evolved usage. Make's original purpose was to allow for defining builds in terms of dependency graphs.
hackly 4y ago
I’m in the unenviable position having to study a build system in Make for a very large codebase. Are there existing tools that generate the dependency graph or aid in understanding such a build system?
- npendleton 4y ago
  Might be worth a look at remake, it can help a ton when debugging complex Make setups: https://github.com/rocky/remake https://github.com/rocky/remake
  - hackly 4y ago
    Very helpful. Thanks so much!
- deleted 4y ago
  [deleted]
amelius 4y ago
Pretty happy with ccache, though of course it doesn't scale to very large projects.
svilen_dobrev 4y ago
the list of inconveniences is missing the biggest (and sadly, unchangeable) pain-point of Make: targets/deps/filenames cannot have spaces in them.. and, it cannot have multiple wildcard-patterns, like %xyz%, only single %xyzabc or abcxyz%, and this brings much complicatedness into picture..
- dundarious 4y ago
  Knit supports regex rule patterns so I assume the latter issue is resolved in knit in one form or another.
- zyedidia 4y ago
  I haven't run into these pain points much with Make, but Knit does support spaces in targets/prerequisites (use single or double quotes to surround the name). For the second point, Knit has regex rules (a Plan9 Mk feature) that can allow multiple grouped matches. The example from the article is: $ ($name-(.*)-(.*)):RB: GOOS=$match2 GOARCH=$match3 go build -o $match1 which is a metarule that matches targets like `$name-linux-amd64` and can be used for easy cross compilation with Go. Of course since it allows full regular expressions, even more complicated rules are allowed.
brvier 4y ago
[flagged]
ledauphin 4y ago
I'm pleased to see the inclusion of hash-based cache invalidation!
c-smile 4y ago
I think it makes sense to combine this with premake (https://premake.github.io/ https://premake.github.io/) Premake is also Lua based and while it does slightly different thing (generates VS,Xcode,gmake solutions) it may benefit from having direct build path - without creating make files and intermediate solutions.
blondin 4y ago
ah, what a missed opportunity for calling this build system katana. or shuriken. it would have continued the analogy that started with ninja and samurai :)
oftenwrong 4y ago
This project has an impressive scope, especially for a personal project. I think there is a strong demand for a small-scale build system like make, but without its footguns. I will be keeping an eye on this.
AceJohnny2 4y ago
I wonder if the author has read the paper "Build Systems A La Carte" [1], which defines a taxonomy of build systems (of which Excel is part!) and their fundamental concepts. This: > I have to use hacks like .PHONY to mark rules as virtual. makes me think not (.PHONY is required because fundamentally Make uses the filesystem as the "Store", or cache, for what is built, and rules just define Store targets) It's been 5 years since the paper, and I'm still hoping for a a production-ready project that leverages its lessons to create an agnostic, distributed build system. Nevertheless, excellent work in Knit! I'm always happy to see interest in the build system space. I particularly appreciate the use of Lua as the scripting layer. Make's scripting layer is certainly... limited. Also, great to see consideration for generated dependencies (like extracted header files)! [1] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
- AceJohnny2 4y ago
  Oh and tangentially, because it's amusing: why does Make use tabs? Because the original Lex was troublesome and it was convenient in the prototype. Then, there were dozens of users and it was too late to change! https://beebo.org/haycorn/2015-04-20_tabs-and-makefiles.html https://beebo.org/haycorn/2015-04-20_tabs-and-makefiles.html
- zyedidia 4y ago
  Thanks! Perhaps I shouldn't have characterized phony rules as a hack, but just that I think it is cleaner to express that information directly in the rule rather than in a separate rule. The paper you link is great, and I should read it in more depth -- are there any particular lessons or insights from it that you think Knit or other build systems could benefit from?
  - AceJohnny2 4y ago
    It's been a while, but one of the more powerful insights I remember, in this era of compute farms, is abstracting out the Store to be cloud-aware (kinda like ccache, but integrated in the builder. I think Bazel does this?)
IshKebab 4y ago
This definitely fixes a few "micro" problems with Make, but I think most people use Make these days for the same reason they use Bash - not because it's good, but because it's probably already installed. If you're going to require installing another tool why not use one that fixes the "macro" problems too? Notably hermetic builds and dynamic dependencies. E.g. Ninja has dynamic dependencies (IIRC). Bazel et al, and possibly Landlock Make have hermetic builds. The recently announced Buck 2 has both.
nanobowers 4y ago
IMO, Rake was the other killer app for Ruby, but Rails sucked all the oxygen out of the room. It's highly underrated as a both a generic build system and a Make replacement.

87 points by signa11 4y ago

kensai 4y ago

Nice project. Is it ready for prime time? Btw, are there any other alternatives to be compared with it and Make?

lantry 4y ago
I've seen a few, but the only one I can remember off the top of my head is just[0]. [0]: https://github.com/casey/just https://github.com/casey/just
- sontek 4y ago
  I use just and it is amazing.
- ekidd 4y ago
  Yup! Two well-established alternatives are "rake", in the Ruby community, and "just" in the Rust community. Rake is fully programmable in Ruby. Just is a bit less flexible, but it doesn't require learning Ruby, and it's quite pleasant to use. https://ruby.github.io/rake/ https://ruby.github.io/rake/
  - lloeki 4y ago
    Also, many people using rake already are often biased by how Rails uses it, but it's much more than that, and can easily be used as a make replacement, complete with freshness checks and rules. There was an article specifically avout that on the interwebs but I can't find it anymore. These ones cover some bits as well, although they're a tad heavier on using Ruby code. https://martinfowler.com/articles/rake.html https://martinfowler.com/articles/rake.html http://www.throwtheswitch.org/build/rake http://www.throwtheswitch.org/build/rake
- masklinn 4y ago
  It should be noted that just specifically does something make was not designed for but is often conscripted into: it’s not a build tool, it’s a command runner. So when you run a command it runs the dependencies then the commands, in full, every time. I very much prefer that usually, because either the underlying tooling handles checking if all’s right or there is no such checking to do and it would be a problem. But if you expect just to build your C project and handle not touching stuff which doesn’t need to be, you’ll be disappointed.
summarity 4y ago
The dependency resolution is core to https://gittup.org/tup/ https://gittup.org/tup/ which had been adopted a bit at the time it came out but since faded back into obscurity.
- NeutralForest 4y ago
  This looks cool, how have I never seen it mentioned before
usual_science 4y ago
There is build2 https://build2.org/ https://build2.org/
raja_mukherji 4y ago
I've also written my own alternative to make called Rabs (https://github.com/wrapl/rabs https://github.com/wrapl/rabs) which uses an imperative language to specify the dependency graph as well as the build instructions. It's designed for really fast incremental builds.

jjgreen 4y ago

Looks OK but adoption will be limited by the pain felt by the users of packages by people that use it. Ah, this package looks nice, I'll have a look. What's a Knitfile? Oh, that looks quite cool; oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it. Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ...

deleted 4y ago
[deleted]
pxc 4y ago
> Ah, this package looks nice, I'll have a look. What's a Knitfile? Oh, that looks quite cool; oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it. Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ... This is a useful vignette, but I wanna pick at it a bit because at least sometimes, I don't see things going this way with me. > oh, so I need to install knit of course. Prebuilt binaries from the internet? Ha ha, no, better to build it Do you really see most devs thinking this? I don't, even though I'm someone who would build it myself. > Oh it's in Go, so I need to install Go? Ah, a different package that does the same thing ... I don't understand what the problem would be here. I maintain several out-of-tree Go packages for Nix for private use and all of them are trivial because Go packages are super easy to build. Are there really many devs out there who are so hesitant to build Go projects, of packages in all languages? With C and autotools you may already have the toolchain installed, but it's generally harder to build than Go packages.
janslow 4y ago
I try to use Make to bootstrap project-specific tooling when missing, by getting it to download (then verify) the tools, then delegate everything else to something like knit (or just/cargo/npm). That way, there's very few prerequisites for a new dev to install (plus you have consistent build/test/run targets for all projects). If someone wants to manage tools themselves (e.g, the no prebuilt binaries example), they can do that, then skip the Makefile entirely.
zyedidia 4y ago

aabdelhafez 4y ago

How does it compare to just? https://github.com/casey/just https://github.com/casey/just

neerajk 4y ago
compare to just what? what? just what? just what?
oftenwrong 4y ago
Knit, like Make, is designed to provide incremental builds. In other words, a build that takes into account the graph of dependencies involved. Their goal is to only run build steps that need to be run. If a build target has already been built, and its dependencies have not changed, then the build rules associated with the target can be skipped. Just, as far as I know, does not support incremental builds. Just is more intended to provide a set of project-specific scripts that are convenient to list and run. Make is sometimes used in that way with "phony" targets, but that is more of an evolved usage. Make's original purpose was to allow for defining builds in terms of dependency graphs.

hackly 4y ago

I’m in the unenviable position having to study a build system in Make for a very large codebase. Are there existing tools that generate the dependency graph or aid in understanding such a build system?

npendleton 4y ago
Might be worth a look at remake, it can help a ton when debugging complex Make setups: https://github.com/rocky/remake https://github.com/rocky/remake
- hackly 4y ago
  Very helpful. Thanks so much!
deleted 4y ago
[deleted]

amelius 4y ago

Pretty happy with ccache, though of course it doesn't scale to very large projects.

svilen_dobrev 4y ago

the list of inconveniences is missing the biggest (and sadly, unchangeable) pain-point of Make: targets/deps/filenames cannot have spaces in them.. and, it cannot have multiple wildcard-patterns, like %xyz%, only single %xyzabc or abcxyz%, and this brings much complicatedness into picture..

dundarious 4y ago
Knit supports regex rule patterns so I assume the latter issue is resolved in knit in one form or another.
zyedidia 4y ago
I haven't run into these pain points much with Make, but Knit does support spaces in targets/prerequisites (use single or double quotes to surround the name). For the second point, Knit has regex rules (a Plan9 Mk feature) that can allow multiple grouped matches. The example from the article is: $ ($name-(.*)-(.*)):RB: GOOS=$match2 GOARCH=$match3 go build -o $match1 which is a metarule that matches targets like `$name-linux-amd64` and can be used for easy cross compilation with Go. Of course since it allows full regular expressions, even more complicated rules are allowed.

brvier 4y ago

[flagged]

ledauphin 4y ago

I'm pleased to see the inclusion of hash-based cache invalidation!

c-smile 4y ago

I think it makes sense to combine this with premake (https://premake.github.io/ https://premake.github.io/) Premake is also Lua based and while it does slightly different thing (generates VS,Xcode,gmake solutions) it may benefit from having direct build path - without creating make files and intermediate solutions.

blondin 4y ago

ah, what a missed opportunity for calling this build system katana. or shuriken. it would have continued the analogy that started with ninja and samurai :)

oftenwrong 4y ago

This project has an impressive scope, especially for a personal project. I think there is a strong demand for a small-scale build system like make, but without its footguns. I will be keeping an eye on this.

AceJohnny2 4y ago

I wonder if the author has read the paper "Build Systems A La Carte" [1], which defines a taxonomy of build systems (of which Excel is part!) and their fundamental concepts. This: > I have to use hacks like .PHONY to mark rules as virtual. makes me think not (.PHONY is required because fundamentally Make uses the filesystem as the "Store", or cache, for what is built, and rules just define Store targets) It's been 5 years since the paper, and I'm still hoping for a a production-ready project that leverages its lessons to create an agnostic, distributed build system. Nevertheless, excellent work in Knit! I'm always happy to see interest in the build system space. I particularly appreciate the use of Lua as the scripting layer. Make's scripting layer is certainly... limited. Also, great to see consideration for generated dependencies (like extracted header files)! [1] https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

AceJohnny2 4y ago
Oh and tangentially, because it's amusing: why does Make use tabs? Because the original Lex was troublesome and it was convenient in the prototype. Then, there were dozens of users and it was too late to change! https://beebo.org/haycorn/2015-04-20_tabs-and-makefiles.html https://beebo.org/haycorn/2015-04-20_tabs-and-makefiles.html
zyedidia 4y ago
Thanks! Perhaps I shouldn't have characterized phony rules as a hack, but just that I think it is cleaner to express that information directly in the rule rather than in a separate rule. The paper you link is great, and I should read it in more depth -- are there any particular lessons or insights from it that you think Knit or other build systems could benefit from?
- AceJohnny2 4y ago
  It's been a while, but one of the more powerful insights I remember, in this era of compute farms, is abstracting out the Store to be cloud-aware (kinda like ccache, but integrated in the builder. I think Bazel does this?)

IshKebab 4y ago

This definitely fixes a few "micro" problems with Make, but I think most people use Make these days for the same reason they use Bash - not because it's good, but because it's probably already installed. If you're going to require installing another tool why not use one that fixes the "macro" problems too? Notably hermetic builds and dynamic dependencies. E.g. Ninja has dynamic dependencies (IIRC). Bazel et al, and possibly Landlock Make have hermetic builds. The recently announced Buck 2 has both.

nanobowers 4y ago

IMO, Rake was the other killer app for Ruby, but Rails sucked all the oxygen out of the room. It's highly underrated as a both a generic build system and a Make replacement.