9 ms·

Show HN: Minimal build system using just /bin/sh

100 points by akkartik 9y ago

lauretas 9y ago
"notabug.org"... interesting. Anyway, how does this compare, features-wise, with GNU Make? For example with make you have `./configure` files, do I also have them here?
- beojan 9y ago
  Configure scripts are part of autotools, not make.
- gardnr 9y ago
  Looking as the sh code, it appears that it will work without tab characters. https://stackoverflow.com/a/2131227/1144060 https://stackoverflow.com/a/2131227/1144060 So there's that.
  - akkartik 9y ago
    As it happens, I was moved to create this standalone repo showing off my current build setup after writing this comment: https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me/djsaxwr/?context=3 https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me... So yes, eliminating tabs (or rather the execrable requirement of using tabs) was absolutely a prime goal.
    - Symbiote 9y ago
      Put this in ~/.editorconfig [Makefile] indent_style = tab And install the plugin, which is available for most editors and IDEs. http://editorconfig.org/ http://editorconfig.org/
    - falcolas 9y ago
      It's not a requirement - I give you: .RECIPEPREFIX > If you prefer to prefix your recipes with a character other than tab, you can set the .RECIPEPREFIX variable to an alternate character That said, how hard is it really to disable tab expansion in your editor? I have a block of configuration dedicated to the different indentation requirements of different programming and configuration languages. In more modern editors, I don't even have to do anything at all to edit Makefiles with tabs - they come pre-configured to do the right thing.
- akkartik 9y ago
  The whole goal of this approach is to get out of the vicious cycle of build files generating other build files, so no `configure`. `configure` is not in itself a feature. What purpose does it fulfill for you? Then we can discuss how we may serve that purpose in some more lightweight manner.
  - blux 9y ago
    I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'. This works in some cases, but certainly not all of them. configure is one of them. Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that.
    - akkartik 9y ago
      "I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'." You're certainly right about that: http://akkartik.name/about http://akkartik.name/about http://akkartik.name/post/libraries2 http://akkartik.name/post/libraries2 https://lobste.rs/s/to8wpr/configuration_files_are_canary_warning https://lobste.rs/s/to8wpr/configuration_files_are_canary_wa... :) "This works in some cases, but certainly not all of them." This too I'm totally willing to accept. I believe most abstractions are prematurely frozen and so poorly designed. But this makes me liable to err too far in the other direction. "configure is one of them. Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that." Here we part ways. I think `configure` is one of the easiest cases for me to be sure of, and it's because I don't care about "all previously supported platforms". Many if not all of them are obsolete. Also, as rossy pointed out elsewhere in this thread (https://news.ycombinator.com/item?id=15045076 https://news.ycombinator.com/item?id=15045076), it's impossible to know today if there's a bug in autotools for some rare platform. We live in a world very different from the one autotools were built for, and it's almost a monoculture at this point. All that crap is utterly unnecessary today. I don't care about building software on all possible platforms, all I care about is building it on this one platform in front of me right now. The most frustrating thing about autotools is that it's unclear which platform each individual check is concerned with. I have a whole mess of rules from 20 years ago -- and zero rationale for any of the rules. That makes even the few rules that are worth having a net liability because I can't separate them from all the useless ones.
- majewsky 9y ago
  > For example with make you have `./configure` files, do I also have them here? The concept of `./configure` is 100% orthogonal to Makefiles. You can have Makefiles without `./configure`, and you can have `./configure` that outputs something other than a Makefile. For example, I could easily imagine a `./configure` that prints a shell script like the one in the submission.
boomlinde 9y ago
This leaves the topological sorting of the tasks to the programmer? This is neat, but that's one very useful feature of make that's missing.
- akkartik 9y ago
  Yes, that's true. I'm still noodling on that, primarily to enable parallel builds. Outside of parallel builds, however, I actually prefer an explicit sequence of steps to so-called "declarative syntax". The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway. And it's better for the reader as well to see the commands laid out in order, when they need to debug breakage on their setup. People should stop pretending to be Vulcans and just write a script for building their project, and then sprinkle in `older_than` directives as an optional optimization. Building from scratch should be the case to optimize for readability, rather than some exceptional situation. [1] https://github.com/zetavm/zetavm/commit/4e5e2e9d8b https://github.com/zetavm/zetavm/commit/4e5e2e9d8b [2] https://github.com/tekknolagi/stackx/commit/5999202423 https://github.com/tekknolagi/stackx/commit/5999202423 [3] Fun fact: when you try to build OpenBSD's userland from source, the top-level Makefile always unconditionally calls `make clean`. I've seen this pattern a few times: large projects find themselves up shit creek in the build system, and it's difficult enough to test and debug the dependency graph that they give up all performance and retreat to just recompiling everything, just to be safe.
  - vog 9y ago
    > The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway This is why the more elaborate build systems recognize dependencies automatically. For Makefiles, usually some sub-Makefile is generated automatically and then included. For language-specific build systems like OPAM (OCaml) or Cargo (Rust), this is one of their core features: As a programmer, don't care about maintaining dependency definitions! Whenever a source file is changed, the build system recalculates the relevant part of the dependency graph as well.
vog 9y ago
I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error handling. (yes, I'm aware of "set -eu") Also, if you occationally need some more elaborated data structures or algorithms, you can express these in a straight forward way, rather than a series of complicated (and usually very fragile) text processing steps.
- akkartik 9y ago
  Totally agreed; I'd use a different language if I needed even an array.
- adtac 9y ago
  I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.
  - vog 9y ago
    If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph). You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/ http://gittup.org/tup/ And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.
    - akkartik 9y ago
      Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.
    - chenglou 9y ago
jlebrech 9y ago
has anyone tried tup? http://gittup.org/tup/ http://gittup.org/tup/
jwilk 9y ago
> This repo illustrates a way to manage a project using just plain Bourne sh, available on all Unix systems since 1977. No, this code uses functions, which didn't exist in the original Bourne shell. They were introduced only in 1984. Source: https://www.in-ulm.de/~mascheck/bourne/#variants https://www.in-ulm.de/~mascheck/bourne/#variants
- akkartik 9y ago
  Thanks for the correction!
- jperkin 9y ago
  It also uses the "local" keyword, which is non-portable. People like to bash the autotools, but they exist for exactly this reason - they already spent decades working through all these gritty details to ensure the scripts generated are portable. To paraphrase Spencer "Those who do not understand autotools are doomed to re-invent them, poorly."
  - akkartik 9y ago
    Uh-uh. Pray tell me an actual extant platform without `local`. Debian dash deems 3 POSIX extensions basic enough to be in /bin/sh -- including `local`. https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts https://www.debian.org/doc/debian-policy/ch-files.html#s-scr... I don't care about some spherical-cow ideal of portability. It has its own costs -- like autotools. This is why OP makes no mention of POSIX or portability. (This exchange regurgitates https://www.reddit.com/r/tinycode/comments/6md2l8/make_gmake_cmake_ninja_minimal_build_system_in_50/dk0xsq9 https://www.reddit.com/r/tinycode/comments/6md2l8/make_gmake..., albeit with more mutual sneering.)
    - jperkin 9y ago
      Any OS which use the Korn shell as /bin/sh, for example Solaris 10, 11, and illumos: $ /bin/sh -c "fail() { local f; }; fail" /bin/sh[1]: local: not found [No such file or directory] $ /bin/sh --version version sh (AT&T Research) 93t+ 2010-03-05 $ uname -rsv SunOS 5.11 joyent_20160721T174127Z It wouldn't work on Solaris 9 and older either as the original /bin/sh also doesn't support "local", but as those releases are EOL it's fine to discount them. Whether you regard these platforms as extant depends on your point of view. Most people wouldn't, and that's fine. We'd naturally disagree ;)
zimpenfish 9y ago
How does this compare to e.g. @apenwarr's minimal `do`?
reacweb 9y ago
One of the benefits I see for this approach is the possibility to handle spaces in filenames (AFAIK very difficult with gnumake). Sadly the provided function does not accept spaces in filenames.
ejholmes 9y ago
A true build system really needs to build out a DAG of dependencies to be efficient and fast. Compose this with https://github.com/ejholmes/walk https://github.com/ejholmes/walk and you're good.
- qznc 9y ago
  Thus ninja https://ninja-build.org/ https://ninja-build.org/
edejong 9y ago
I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA). No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG, you're constantly traversing from one side to the other, making the build 'work'.
- antibjarne 9y ago
  Maybe they're not using a language with insanely bloated compile times?
  - edejong 9y ago
    It's not just about compile time. Couple of examples: - Parallel downloading / uploading of artifacts - Multi-step transformation (old-style: lexer to C to Object file to linker to deployment). You can turn this around, but generally you want each output of your lexer to be input to the C compiler and so on. - Testing (especially brute-force testing might take time) And in modern situations you have the same, with Typescript => Javascript => Minimisation => Combination => Deployment (roughly). It really helps to think of this as a dependency chain and use a partial-order planner to find a possible linearization and execute.
- brudgers 9y ago
  There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository. Alternatives are not replacements.
  - edejong 9y ago
    Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones. A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, CMake, Maven, SBT and many others.
sgt 9y ago
Somehow I find it slightly amusing that I see ./a.out in 2017.
TheDong 9y ago
> The build script is easy to repurpose to your needs No, it's not. This isn't under a free or open source license. The lack of license means that only an idiot would use it for anything.
pskocik 9y ago
I'm also using (for now) something based on these time checks (also pure posix + local, but local is extremely portable (tests well on 7 shells), though mine's a little more involved (automatic build job timing, controlled concurrency, robust failure/cancellation with autodeletion, fancy logging, filterable implicit dependencies, dependency + counting). It scales well to about 1000 source files. Then the lag starts to feel noticable.
sarabasem 9y ago
Watch a movie Total Recall Full Online Translator http://www.saraworldnewsa.ml/2017/08/total-recall.html http://www.saraworldnewsa.ml/2017/08/total-recall.html

100 points by akkartik 9y ago

"notabug.org"... interesting. Anyway, how does this compare, features-wise, with GNU Make? For example with make you have `./configure` files, do I also have them here?

beojan 9y ago
Configure scripts are part of autotools, not make.
gardnr 9y ago
Looking as the sh code, it appears that it will work without tab characters. https://stackoverflow.com/a/2131227/1144060 https://stackoverflow.com/a/2131227/1144060 So there's that.
- akkartik 9y ago
  As it happens, I was moved to create this standalone repo showing off my current build setup after writing this comment: https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me/djsaxwr/?context=3 https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me... So yes, eliminating tabs (or rather the execrable requirement of using tabs) was absolutely a prime goal.
  - Symbiote 9y ago
    Put this in ~/.editorconfig [Makefile] indent_style = tab And install the plugin, which is available for most editors and IDEs. http://editorconfig.org/ http://editorconfig.org/
  - falcolas 9y ago
    It's not a requirement - I give you: .RECIPEPREFIX > If you prefer to prefix your recipes with a character other than tab, you can set the .RECIPEPREFIX variable to an alternate character That said, how hard is it really to disable tab expansion in your editor? I have a block of configuration dedicated to the different indentation requirements of different programming and configuration languages. In more modern editors, I don't even have to do anything at all to edit Makefiles with tabs - they come pre-configured to do the right thing.
akkartik 9y ago
The whole goal of this approach is to get out of the vicious cycle of build files generating other build files, so no `configure`. `configure` is not in itself a feature. What purpose does it fulfill for you? Then we can discuss how we may serve that purpose in some more lightweight manner.
- blux 9y ago
  I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'. This works in some cases, but certainly not all of them. configure is one of them. Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that.
  - akkartik 9y ago
    "I have the feeling your way of thinking is 'remove every abstraction so that we end up with a simpler system'." You're certainly right about that: http://akkartik.name/about http://akkartik.name/about http://akkartik.name/post/libraries2 http://akkartik.name/post/libraries2 https://lobste.rs/s/to8wpr/configuration_files_are_canary_warning https://lobste.rs/s/to8wpr/configuration_files_are_canary_wa... :) "This works in some cases, but certainly not all of them." This too I'm totally willing to accept. I believe most abstractions are prematurely frozen and so poorly designed. But this makes me liable to err too far in the other direction. "configure is one of them. Why? Try removing a configure script for even a simple autotools project, and ensure that it still builds fine on all previously supported platforms by writing a cross platform Makefile. Good luck with that." Here we part ways. I think `configure` is one of the easiest cases for me to be sure of, and it's because I don't care about "all previously supported platforms". Many if not all of them are obsolete. Also, as rossy pointed out elsewhere in this thread (https://news.ycombinator.com/item?id=15045076 https://news.ycombinator.com/item?id=15045076), it's impossible to know today if there's a bug in autotools for some rare platform. We live in a world very different from the one autotools were built for, and it's almost a monoculture at this point. All that crap is utterly unnecessary today. I don't care about building software on all possible platforms, all I care about is building it on this one platform in front of me right now. The most frustrating thing about autotools is that it's unclear which platform each individual check is concerned with. I have a whole mess of rules from 20 years ago -- and zero rationale for any of the rules. That makes even the few rules that are worth having a net liability because I can't separate them from all the useless ones.
majewsky 9y ago
> For example with make you have `./configure` files, do I also have them here? The concept of `./configure` is 100% orthogonal to Makefiles. You can have Makefiles without `./configure`, and you can have `./configure` that outputs something other than a Makefile. For example, I could easily imagine a `./configure` that prints a shell script like the one in the submission.

boomlinde 9y ago

This leaves the topological sorting of the tasks to the programmer? This is neat, but that's one very useful feature of make that's missing.

akkartik 9y ago
Yes, that's true. I'm still noodling on that, primarily to enable parallel builds. Outside of parallel builds, however, I actually prefer an explicit sequence of steps to so-called "declarative syntax". The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway. And it's better for the reader as well to see the commands laid out in order, when they need to debug breakage on their setup. People should stop pretending to be Vulcans and just write a script for building their project, and then sprinkle in `older_than` directives as an optional optimization. Building from scratch should be the case to optimize for readability, rather than some exceptional situation. [1] https://github.com/zetavm/zetavm/commit/4e5e2e9d8b https://github.com/zetavm/zetavm/commit/4e5e2e9d8b [2] https://github.com/tekknolagi/stackx/commit/5999202423 https://github.com/tekknolagi/stackx/commit/5999202423 [3] Fun fact: when you try to build OpenBSD's userland from source, the top-level Makefile always unconditionally calls `make clean`. I've seen this pattern a few times: large projects find themselves up shit creek in the build system, and it's difficult enough to test and debug the dependency graph that they give up all performance and retreat to just recompiling everything, just to be safe.
- vog 9y ago
  > The frequency with which I find bugs in the dependency graph of `Makefile`s [1][2][3] suggests that people are starting from a sequence of commands anyway This is why the more elaborate build systems recognize dependencies automatically. For Makefiles, usually some sub-Makefile is generated automatically and then included. For language-specific build systems like OPAM (OCaml) or Cargo (Rust), this is one of their core features: As a programmer, don't care about maintaining dependency definitions! Whenever a source file is changed, the build system recalculates the relevant part of the dependency graph as well.

vog 9y ago

I applaud the effort, but is "pure /bin/sh" really a good feature in today's systems? To my knowledge, today every Unix system has Python, usually at least 2.7, if not Python 3. Some even replaced most of their system shell scripts with Python scripts. (Formerly, Perl had this status, but never really caught on in that area.) Python is a much cleaner language than any shell language, especially with regard to error handling. (yes, I'm aware of "set -eu") Also, if you occationally need some more elaborated data structures or algorithms, you can express these in a straight forward way, rather than a series of complicated (and usually very fragile) text processing steps.

akkartik 9y ago
Totally agreed; I'd use a different language if I needed even an array.
adtac 9y ago
I'm having a hard time thinking of a use case for complex data structures in a build script. Can you give a real world example? Array is the only one that I have found lacking, in my experience.
- vog 9y ago
  If you aim for slightly larger projects and efficiency, you'd need at least a DAG (because that's the fundamental nature of a dependency graph). You can find a nice explanation in the "tup" build tool, although there are of course many others: http://gittup.org/tup/ http://gittup.org/tup/ And yes, you can express DAGs with text and/or filesystem operations (plus symlinks), but not efficiently and writing code that operates on that level is cumbersome.
  - akkartik 9y ago
    Ah, thanks for the pointer to `tup`! I'd spent a while looking for it a few months ago but couldn't remember the name.
  - chenglou 9y ago

jlebrech 9y ago

has anyone tried tup? http://gittup.org/tup/ http://gittup.org/tup/

jwilk 9y ago

> This repo illustrates a way to manage a project using just plain Bourne sh, available on all Unix systems since 1977. No, this code uses functions, which didn't exist in the original Bourne shell. They were introduced only in 1984. Source: https://www.in-ulm.de/~mascheck/bourne/#variants https://www.in-ulm.de/~mascheck/bourne/#variants

akkartik 9y ago
Thanks for the correction!
jperkin 9y ago
It also uses the "local" keyword, which is non-portable. People like to bash the autotools, but they exist for exactly this reason - they already spent decades working through all these gritty details to ensure the scripts generated are portable. To paraphrase Spencer "Those who do not understand autotools are doomed to re-invent them, poorly."
- akkartik 9y ago
  Uh-uh. Pray tell me an actual extant platform without `local`. Debian dash deems 3 POSIX extensions basic enough to be in /bin/sh -- including `local`. https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts https://www.debian.org/doc/debian-policy/ch-files.html#s-scr... I don't care about some spherical-cow ideal of portability. It has its own costs -- like autotools. This is why OP makes no mention of POSIX or portability. (This exchange regurgitates https://www.reddit.com/r/tinycode/comments/6md2l8/make_gmake_cmake_ninja_minimal_build_system_in_50/dk0xsq9 https://www.reddit.com/r/tinycode/comments/6md2l8/make_gmake..., albeit with more mutual sneering.)
  - jperkin 9y ago
    Any OS which use the Korn shell as /bin/sh, for example Solaris 10, 11, and illumos: $ /bin/sh -c "fail() { local f; }; fail" /bin/sh[1]: local: not found [No such file or directory] $ /bin/sh --version version sh (AT&T Research) 93t+ 2010-03-05 $ uname -rsv SunOS 5.11 joyent_20160721T174127Z It wouldn't work on Solaris 9 and older either as the original /bin/sh also doesn't support "local", but as those releases are EOL it's fine to discount them. Whether you regard these platforms as extant depends on your point of view. Most people wouldn't, and that's fine. We'd naturally disagree ;)

zimpenfish 9y ago

How does this compare to e.g. @apenwarr's minimal `do`?

reacweb 9y ago

One of the benefits I see for this approach is the possibility to handle spaces in filenames (AFAIK very difficult with gnumake). Sadly the provided function does not accept spaces in filenames.

ejholmes 9y ago

A true build system really needs to build out a DAG of dependencies to be efficient and fast. Compose this with https://github.com/ejholmes/walk https://github.com/ejholmes/walk and you're good.

qznc 9y ago
Thus ninja https://ninja-build.org/ https://ninja-build.org/

edejong 9y ago

I seriously wonder if the author ever worked on a project with more than around 10k LOC, multiple languages and tools, automatic testing, deployment, version control (merging this build script is going to be a huge PITA). No, you have to realize you're working on one side of the graph (source code), but your build produces something on the other side (your artifact). Without automatic dependency resolution of you DAG, you're constantly traversing from one side to the other, making the build 'work'.

antibjarne 9y ago
Maybe they're not using a language with insanely bloated compile times?
- edejong 9y ago
  It's not just about compile time. Couple of examples: - Parallel downloading / uploading of artifacts - Multi-step transformation (old-style: lexer to C to Object file to linker to deployment). You can turn this around, but generally you want each output of your lexer to be input to the C compiler and so on. - Testing (especially brute-force testing might take time) And in modern situations you have the same, with Typescript => Javascript => Minimisation => Combination => Deployment (roughly). It really helps to think of this as a dependency chain and use a partial-order planner to find a possible linearization and execute.
brudgers 9y ago
There are a lot of projects that don't meet those criteria. Scale is bidirectional. Tools designed for big projects can have features and designs that don't scale down well. Tools designed for big projects often depend on institutional type knowledge where expertise is spread among multiple individuals...at Google, there teams of engineers managing the single code tree is what allows a monorepository. Alternatives are not replacements.
- edejong 9y ago
  Yes, it's a team of engineers. But interestingly, Google started with a mono-repo. All through their explosive growth they maintained (mostly, except Android and Chromium) their vision of mono-repo. It clearly shows that their vision of mono-repo works both for small companies, as well as large ones. A build-script without dependency resolution clearly does not scale. It was the reason to design and implement Make, CMake, Maven, SBT and many others.

sgt 9y ago

Somehow I find it slightly amusing that I see ./a.out in 2017.

TheDong 9y ago

> The build script is easy to repurpose to your needs No, it's not. This isn't under a free or open source license. The lack of license means that only an idiot would use it for anything.

pskocik 9y ago

I'm also using (for now) something based on these time checks (also pure posix + local, but local is extremely portable (tests well on 7 shells), though mine's a little more involved (automatic build job timing, controlled concurrency, robust failure/cancellation with autodeletion, fancy logging, filterable implicit dependencies, dependency + counting). It scales well to about 1000 source files. Then the lag starts to feel noticable.

sarabasem 9y ago

Watch a movie Total Recall Full Online Translator http://www.saraworldnewsa.ml/2017/08/total-recall.html http://www.saraworldnewsa.ml/2017/08/total-recall.html