7 ms·

Show HN: Hancho – A simple and pleasant build system in ~500 lines of Python

162 points by aappleby 3y ago

Hi HN, I've been taking a break from my big side projects to work on a smaller side project - a tiny build system that's based on what I've learned from using Ninja and ad-hoc Python for my homebrew build systems over the last few years. It's basically a promise-based dependency graph runner plus a simple text templating engine, and it works quite well for the smallish projects I've tried it out on so far. If you find Make crufty, CMake inconsistent, Ninja verbose, and Bazel just too much build system, give Hancho a try. Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like.

kklikkkk 3y ago
[flagged]
- aappleby 3y ago
  ?
pama 3y ago
Thanks for keeping this project simple and clean in a single file of Python.
- aappleby 3y ago
  You are welcome! Single-file tools please me.
donatj 3y ago
TIL “honcho” is not a Native American term as I’d always assumed, but Japanese? https://www.merriam-webster.com/dictionary/honcho https://www.merriam-webster.com/dictionary/honcho
- xanderlewis 3y ago
  Yes! I was amazed several years ago after I had learnt the word 班長 (hanchō)… and a while later realised it sounded a bit like the phrase ‘head honcho’…and then realised that that’s actually where it came from. There are lot of words ending in 長 (chō) in Japanese that refer to people in positions of leadership: shachō (company director), kōchō (headmaster), kichō (captain, of an aircraft). Another somewhat surprising example of unexpected Japanese etymology: ‘rickshaw’ comes from the Japanese 人力車 (jin-rikisha, literally “person-powered-car”). I would’ve probably expected it to have come from an Indian language, or something.
  - matvore 3y ago
    Some say hunky dory comes from "honcho doori" or 本町通り (personally I don't see how ki can come from cho)
    - xanderlewis 3y ago
      Would be fun if true! Seems to somewhat match semantically as well.
e-dant 3y ago
It’s the integration with those platform-specific requirements which I find so tough. Apple wants a particular format, signed and whatnot via xcodebuild. Windows, I don’t even know. Something about universal apps on the Windows store. Android is a bit simpler if you can target the right directory with the right toolchain. It’s these particular systems that I find the most difficult. If there were a collection of nice utilities for working with them uniformly, oh, that would be amazing.
- catlifeonmars 3y ago
  Not really the same as app packager, but your comment reminded me of this tool: https://fpm.readthedocs.io/en/v1.15.1/ https://fpm.readthedocs.io/en/v1.15.1/
- aappleby 3y ago
  Author here. I do most of my development in WSL but do have a few projects that I want to get working again with Windows/MSVC - I will be using Hancho for those builds and expect it to work fine cross-platform though I haven't tackled it yet.
notso411 3y ago
[dead]
frumiousirc 3y ago
This looks like a nice distillation of core DAG-running features. I really like the simplicity of `honcho` and will likely try it next time I have a DAG to run that fits in to its model. I can not help but compare `honcho` to others in this space that I know, particularly `waf` and `snakemake`. I guess these do cover a larger feature space than `honcho` and that comes at the cost of rather greater complexity. I don't think sacrificing the simplicity of `honcho` to add features is necessarily a good thing but I do wonder if / how `honcho` might be used in these ways: - Execution of a rule adds a new task node to the DAG (a `TaskGen` in `waf`). - Implicit DAG forming and rule execution (akin to how we define a `rule:` for `snakemake` but it is the system that determines what rules to run and then runs them). - DAG edge types besides files. Eg, run a downstream rule if some Python data object's value changes (instead of a file change). I believe neither `waf` nor `snakemake` supports this and but one can serialize the Python data to file to make it fit the DAG engine. - Batteries included such as a cross-platform version of the `c_binary()` function in the `honcho` tutorial. (like `waf` "tools"). - In-system dependency generators (eg `waf` "scanner" pattern). Thanks for sharing `honcho`!
- frumiousirc 3y ago
  PS: please assert an explicit license!
  - Filligree 3y ago
    That assumes it’s supposed to have a license. “Proprietary” is a perfectly reasonable choice.
    - codetrotter 3y ago
      Sure but 1. In that case posting the source code to GitHub is less useful 2. More to the point, op said: “Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like.” I think OP simply forgot to add a license. In which case, reminding them to do so is useful.
      - 3y ago
rcarmo 3y ago
I like it. I wrote Piku (https://github.com/piku/piku https://github.com/piku/piku) with much the same interest in fixing some of my pains, so I get where you're coming from with this. Will drop it into one of my current projects to build ESP32 binaries :)
- aappleby 3y ago
  Please ping me back with your experience using it for ESP32 stuff. I've already used Hancho to build FPGA bitstreams with the Icestorm tools and it was delightful, but as the author I'm not a neutral observer. :)
  - rcarmo 3y ago
    Sure. Right now I'm wrestling with taking an old codebase and getting it to build under PlatformIO with the right dependencies on a new machine. Once it builds once, I'll go and replicate the process under honcho. Edit: if you use RSS, keep an eye on https://taoofmac.com https://taoofmac.com, it will show up there...
- deleted 3y ago
  [deleted]
chubot 3y ago
Hm it seems like the defining feature of Bazel is high level "macros" like cc_library() and proto_library() on top of the low level build graph I looked at the examples and tutorial and it still seems more low level, in that you mention object files and link them? main_o = compile("src/main.cpp", "build/tut1/src/main.o") util_o = compile("src/util.cpp", "build/tut1/src/util.o") link([main_o, util_o], "build/tut1/app") One reason I like the layering is for build variants -- dbg opt ASAN UBSAN -- something Make does very poorly. IMO you don't want to mention literal object file paths in the build config for this reason. I use a tree layout like _build/obj/cxx-asan/frontend.syntax.o _build/obj/cxx-asan/core/process.o _build/obj/cxx-ubsan/frontend.syntax.o _build/obj/cxx-ubsan/core/process.o and then this is abstracted with the Bazel-like target syntax //frontend/syntax //core/process This works well - you don't have to clean when rebuilding variants, and all the object sharing between test binaries really speeds up the build. IMO this is 100% essential for writing C++ these days -- all tests should be run with ASAN and UBSAN. --- I wrote a mini-bazel on top of Ninja with these features: https://www.oilshell.org/blog/2022/10/garbage-collector.html#declarative-ninja-mini-bazel https://www.oilshell.org/blog/2022/10/garbage-collector.html... So it's ~1700 lines, though some of that is our own logic, and extra features like computing preprocessed size, which I used to improve compile times. You get the build macros like asdl_library() generating C++ and Python (the same as proto_library(), a schema language that generates code) And it also correctly finds dependencies of code generators. So if you change a .py file that is imported by another .py file that is used to generated a C++ header, everything will work. That was one of the trickier bits, with Ninja implicit dependencies. This build file example mixes low level Ninja n.rule() and n.build() with high level r.cc_library() and so forth. I find this layering really does make it scale better for bigger projects https://github.com/oilshell/oil/blob/master/asdl/NINJA_subgraph.py https://github.com/oilshell/oil/blob/master/asdl/NINJA_subgr... Some more description - https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tutskb https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tu... Comment about Chrome/Android using the same pattern - https://lobste.rs/s/0qc6vp/arcan_0_6_3_i_pty_fool#c_ahqwky https://lobste.rs/s/0qc6vp/arcan_0_6_3_i_pty_fool#c_ahqwky which is explicitly in the Android docs - https://android.googlesource.com/platform/build/bazel/+/7d96ed30359f12311dd7a78c399e29eb695b2012/docs/concepts.md https://android.googlesource.com/platform/build/bazel/+/7d96... The main thing my system doesn't have as mentioned is the per-action build sandboxing, which Bazel has OS-specific wrappers for. That's really useful for finding missing deps. Right now it's not a big pain point, since the build will break in a slightly less obvious way if you're missing deps.
BiteCode_dev 3y ago
We need more trials in this space, building is still not fun after decades of it. Personally, I think doit (https://www.bitecode.dev/p/doit-the-goodest-python-task-runner https://www.bitecode.dev/p/doit-the-goodest-python-task-runn...) is still the winner in the python space, but there is a lot of it that could be improved.
- aappleby 3y ago
  Author here. I hadn't heard of Doit, but it does look similar to Hancho. Hancho pushes much harder on the minimalism side, however. I think I may steal Doit's "actions can be arrays of commands" feature, as that would be trivial to add to Hancho and potentially useful.
- xyst 3y ago
  I have used pantsbuild in the past. Had a pleasant experience with it.
usrnm 3y ago
I'm looking at the example and I just don't see how it's better than a Makefile. What am I missing?
- aappleby 3y ago
  It's not "better" than a Makefile, just different. I find it easier to understand and maintain. I've used Makefiles for decades and I still find myself looking up basic Make syntax.
- Brian_K_White 3y ago
  Maybe that it never claimed to be better than Make.
samstave 3y ago
Even though I will never likely need/want this personally - I LOVE all the posts and dialogue on this topic... which else woud have never entered my awareness... +1 to all
Terretta 3y ago
Whenever looking at one these, I think back to the obscure but interesting "tup", and try to decide if this new thing is better. Because tup is awesome: “How is it so awesome? In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up.” https://gittup.org/tup/ https://gittup.org/tup/ On a more serious note, the whitepaper: https://gittup.org/tup/build_system_rules_and_algorithms.pdf https://gittup.org/tup/build_system_rules_and_algorithms.pdf
- aappleby 3y ago
  Tup is interesting, but it requires keeping track of dependencies in a SQL (sqlite?) database - already more complex than Hancho it seems.
- deleted 3y ago
  [deleted]
xyst 3y ago
Reminds me of pantsbuild.
meribold 3y ago
> Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like. It's fascinating to think of this as a deliberate approach or paradigm rather than a shortcoming. The antithesis to the inner-platform effect [1]. Instead of digging an ever deeper rabbit hole of options and configuration, encourage customization via forking. You can't beat the flexibility. And lots of projects have well over 500 lines of build system configuration anyway. With that kind of competition, forking a 500-line build system may well yield the more comprehensible end result. Not to mention that when facing a problem outside of a given build system's anticipated set, I'd much rather have Python to solve that with than a special build system language. [1]: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect
- teaearlgraycold 3y ago
  I’m a big fan of copy-pasting code. When you are just learning to program you learn all of the common abstractions, for loops, functions, modules, etc. to reduce code duplication. This leads too many devs to think duplication is a bad thing to be avoided at all costs. I’ve seen a ton of terrible libraries out there that aim to end code duplication. Please, I can handle needing to copy/paste a couple of lines of code around my code base. But I can’t handle your bad abstractions.
  - nine_k 3y ago
    Copy-pasting is fun when you add code, and a pain when you subsequently have to alter it. So it's easy to sell to a beginner or a hobbyist, and it also works well in projects which do not live long, like most games and many websites.
    - smilliken 3y ago
      It can go either way. If you duplicate code, but the two copies need to stay in sync, then you've made it a bit harder and more error-prone to maintain. But the worse offense is if you have some code that serves two conflicting sets of requirements, but you keep the code together with increasing complexity to thread the needle of satisfying all the requirements. The latter is the worse offense because it's much harder to decouple two systems than it is to notice a similarity in two bits of code and to extract the commonality.
kazinator 3y ago
"Hanchō" is the likely origin of "head honcho".
Brian_K_White 3y ago
Thank you for being interesting and useful without trying to claim to be better than grampas old make or claiming to solve all problems.

162 points by aappleby 3y ago

kklikkkk 3y ago

[flagged]

aappleby 3y ago
?

pama 3y ago

Thanks for keeping this project simple and clean in a single file of Python.

aappleby 3y ago
You are welcome! Single-file tools please me.

donatj 3y ago

TIL “honcho” is not a Native American term as I’d always assumed, but Japanese? https://www.merriam-webster.com/dictionary/honcho https://www.merriam-webster.com/dictionary/honcho

xanderlewis 3y ago
Yes! I was amazed several years ago after I had learnt the word 班長 (hanchō)… and a while later realised it sounded a bit like the phrase ‘head honcho’…and then realised that that’s actually where it came from. There are lot of words ending in 長 (chō) in Japanese that refer to people in positions of leadership: shachō (company director), kōchō (headmaster), kichō (captain, of an aircraft). Another somewhat surprising example of unexpected Japanese etymology: ‘rickshaw’ comes from the Japanese 人力車 (jin-rikisha, literally “person-powered-car”). I would’ve probably expected it to have come from an Indian language, or something.
- matvore 3y ago
  Some say hunky dory comes from "honcho doori" or 本町通り (personally I don't see how ki can come from cho)
  - xanderlewis 3y ago
    Would be fun if true! Seems to somewhat match semantically as well.

e-dant 3y ago

It’s the integration with those platform-specific requirements which I find so tough. Apple wants a particular format, signed and whatnot via xcodebuild. Windows, I don’t even know. Something about universal apps on the Windows store. Android is a bit simpler if you can target the right directory with the right toolchain. It’s these particular systems that I find the most difficult. If there were a collection of nice utilities for working with them uniformly, oh, that would be amazing.

catlifeonmars 3y ago
Not really the same as app packager, but your comment reminded me of this tool: https://fpm.readthedocs.io/en/v1.15.1/ https://fpm.readthedocs.io/en/v1.15.1/
aappleby 3y ago
Author here. I do most of my development in WSL but do have a few projects that I want to get working again with Windows/MSVC - I will be using Hancho for those builds and expect it to work fine cross-platform though I haven't tackled it yet.

notso411 3y ago

[dead]

frumiousirc 3y ago

This looks like a nice distillation of core DAG-running features. I really like the simplicity of `honcho` and will likely try it next time I have a DAG to run that fits in to its model. I can not help but compare `honcho` to others in this space that I know, particularly `waf` and `snakemake`. I guess these do cover a larger feature space than `honcho` and that comes at the cost of rather greater complexity. I don't think sacrificing the simplicity of `honcho` to add features is necessarily a good thing but I do wonder if / how `honcho` might be used in these ways: - Execution of a rule adds a new task node to the DAG (a `TaskGen` in `waf`). - Implicit DAG forming and rule execution (akin to how we define a `rule:` for `snakemake` but it is the system that determines what rules to run and then runs them). - DAG edge types besides files. Eg, run a downstream rule if some Python data object's value changes (instead of a file change). I believe neither `waf` nor `snakemake` supports this and but one can serialize the Python data to file to make it fit the DAG engine. - Batteries included such as a cross-platform version of the `c_binary()` function in the `honcho` tutorial. (like `waf` "tools"). - In-system dependency generators (eg `waf` "scanner" pattern). Thanks for sharing `honcho`!

frumiousirc 3y ago
PS: please assert an explicit license!
- Filligree 3y ago
  That assumes it’s supposed to have a license. “Proprietary” is a perfectly reasonable choice.
  - codetrotter 3y ago
    Sure but 1. In that case posting the source code to GitHub is less useful 2. More to the point, op said: “Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like.” I think OP simply forgot to add a license. In which case, reminding them to do so is useful.
    - 3y ago

rcarmo 3y ago

I like it. I wrote Piku (https://github.com/piku/piku https://github.com/piku/piku) with much the same interest in fixing some of my pains, so I get where you're coming from with this. Will drop it into one of my current projects to build ESP32 binaries :)

aappleby 3y ago
Please ping me back with your experience using it for ESP32 stuff. I've already used Hancho to build FPGA bitstreams with the Icestorm tools and it was delightful, but as the author I'm not a neutral observer. :)
- rcarmo 3y ago
  Sure. Right now I'm wrestling with taking an old codebase and getting it to build under PlatformIO with the right dependencies on a new machine. Once it builds once, I'll go and replicate the process under honcho. Edit: if you use RSS, keep an eye on https://taoofmac.com https://taoofmac.com, it will show up there...
deleted 3y ago
[deleted]

chubot 3y ago

Hm it seems like the defining feature of Bazel is high level "macros" like cc_library() and proto_library() on top of the low level build graph I looked at the examples and tutorial and it still seems more low level, in that you mention object files and link them? main_o = compile("src/main.cpp", "build/tut1/src/main.o") util_o = compile("src/util.cpp", "build/tut1/src/util.o") link([main_o, util_o], "build/tut1/app") One reason I like the layering is for build variants -- dbg opt ASAN UBSAN -- something Make does very poorly. IMO you don't want to mention literal object file paths in the build config for this reason. I use a tree layout like _build/obj/cxx-asan/frontend.syntax.o _build/obj/cxx-asan/core/process.o _build/obj/cxx-ubsan/frontend.syntax.o _build/obj/cxx-ubsan/core/process.o and then this is abstracted with the Bazel-like target syntax //frontend/syntax //core/process This works well - you don't have to clean when rebuilding variants, and all the object sharing between test binaries really speeds up the build. IMO this is 100% essential for writing C++ these days -- all tests should be run with ASAN and UBSAN. --- I wrote a mini-bazel on top of Ninja with these features: https://www.oilshell.org/blog/2022/10/garbage-collector.html#declarative-ninja-mini-bazel https://www.oilshell.org/blog/2022/10/garbage-collector.html... So it's ~1700 lines, though some of that is our own logic, and extra features like computing preprocessed size, which I used to improve compile times. You get the build macros like asdl_library() generating C++ and Python (the same as proto_library(), a schema language that generates code) And it also correctly finds dependencies of code generators. So if you change a .py file that is imported by another .py file that is used to generated a C++ header, everything will work. That was one of the trickier bits, with Ninja implicit dependencies. This build file example mixes low level Ninja n.rule() and n.build() with high level r.cc_library() and so forth. I find this layering really does make it scale better for bigger projects https://github.com/oilshell/oil/blob/master/asdl/NINJA_subgraph.py https://github.com/oilshell/oil/blob/master/asdl/NINJA_subgr... Some more description - https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tutskb https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tu... Comment about Chrome/Android using the same pattern - https://lobste.rs/s/0qc6vp/arcan_0_6_3_i_pty_fool#c_ahqwky https://lobste.rs/s/0qc6vp/arcan_0_6_3_i_pty_fool#c_ahqwky which is explicitly in the Android docs - https://android.googlesource.com/platform/build/bazel/+/7d96ed30359f12311dd7a78c399e29eb695b2012/docs/concepts.md https://android.googlesource.com/platform/build/bazel/+/7d96... The main thing my system doesn't have as mentioned is the per-action build sandboxing, which Bazel has OS-specific wrappers for. That's really useful for finding missing deps. Right now it's not a big pain point, since the build will break in a slightly less obvious way if you're missing deps.

BiteCode_dev 3y ago

We need more trials in this space, building is still not fun after decades of it. Personally, I think doit (https://www.bitecode.dev/p/doit-the-goodest-python-task-runner https://www.bitecode.dev/p/doit-the-goodest-python-task-runn...) is still the winner in the python space, but there is a lot of it that could be improved.

aappleby 3y ago
Author here. I hadn't heard of Doit, but it does look similar to Hancho. Hancho pushes much harder on the minimalism side, however. I think I may steal Doit's "actions can be arrays of commands" feature, as that would be trivial to add to Hancho and potentially useful.
xyst 3y ago
I have used pantsbuild in the past. Had a pleasant experience with it.

usrnm 3y ago

I'm looking at the example and I just don't see how it's better than a Makefile. What am I missing?

aappleby 3y ago
It's not "better" than a Makefile, just different. I find it easier to understand and maintain. I've used Makefiles for decades and I still find myself looking up basic Make syntax.
Brian_K_White 3y ago
Maybe that it never claimed to be better than Make.

samstave 3y ago

Even though I will never likely need/want this personally - I LOVE all the posts and dialogue on this topic... which else woud have never entered my awareness... +1 to all

Terretta 3y ago

Whenever looking at one these, I think back to the obscure but interesting "tup", and try to decide if this new thing is better. Because tup is awesome: “How is it so awesome? In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up.” https://gittup.org/tup/ https://gittup.org/tup/ On a more serious note, the whitepaper: https://gittup.org/tup/build_system_rules_and_algorithms.pdf https://gittup.org/tup/build_system_rules_and_algorithms.pdf

aappleby 3y ago
Tup is interesting, but it requires keeping track of dependencies in a SQL (sqlite?) database - already more complex than Hancho it seems.
deleted 3y ago
[deleted]

xyst 3y ago

Reminds me of pantsbuild.

meribold 3y ago

> Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like. It's fascinating to think of this as a deliberate approach or paradigm rather than a shortcoming. The antithesis to the inner-platform effect [1]. Instead of digging an ever deeper rabbit hole of options and configuration, encourage customization via forking. You can't beat the flexibility. And lots of projects have well over 500 lines of build system configuration anyway. With that kind of competition, forking a 500-line build system may well yield the more comprehensible end result. Not to mention that when facing a problem outside of a given build system's anticipated set, I'd much rather have Python to solve that with than a special build system language. [1]: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect

teaearlgraycold 3y ago
I’m a big fan of copy-pasting code. When you are just learning to program you learn all of the common abstractions, for loops, functions, modules, etc. to reduce code duplication. This leads too many devs to think duplication is a bad thing to be avoided at all costs. I’ve seen a ton of terrible libraries out there that aim to end code duplication. Please, I can handle needing to copy/paste a couple of lines of code around my code base. But I can’t handle your bad abstractions.
- nine_k 3y ago
  Copy-pasting is fun when you add code, and a pain when you subsequently have to alter it. So it's easy to sell to a beginner or a hobbyist, and it also works well in projects which do not live long, like most games and many websites.
  - smilliken 3y ago
    It can go either way. If you duplicate code, but the two copies need to stay in sync, then you've made it a bit harder and more error-prone to maintain. But the worse offense is if you have some code that serves two conflicting sets of requirements, but you keep the code together with increasing complexity to thread the needle of satisfying all the requirements. The latter is the worse offense because it's much harder to decouple two systems than it is to notice a similarity in two bits of code and to extract the commonality.

kazinator 3y ago

"Hanchō" is the likely origin of "head honcho".

Brian_K_White 3y ago

Thank you for being interesting and useful without trying to claim to be better than grampas old make or claiming to solve all problems.