7 ms·

Proposal to add build graph output to GNU Make (2020)

161 points by jonasdn 5y ago

pjmlp 5y ago
Some clarification needed, > The Android build system works by including all recipes to be built (programs / libraries / etc) using the GNU Make include directive, so that you end up with one giant Makefile that holds all rules for building the platform. Possibly to avoid the problems laid out in the paper Recursive make considered harmful. The old build system (ndk-build) works like this. Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong.
- darkamaul 5y ago
  > Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong. Soong build files are easy to parse and you can create an Unified Dependency Graph from them. That's what I did in BGraph [0] for security applications. [0] https://github.com/quarkslab/bgraph https://github.com/quarkslab/bgraph
- malkia 5y ago
  AFAIK, Soong -> Bazel - https://blog.bazel.build/2020/11/12/aosp_migrating_to_bazel.html https://blog.bazel.build/2020/11/12/aosp_migrating_to_bazel.... (2020)
kergonath 5y ago
This would be quite useful. We have a couple of large-ish projects with a couple of hundreds files compiled in parallel and there are snags every now and then (which disappear when re-running make either in parallel or sequential modes). Getting a graph in these cases would be a nice time saver. I’ll try the fork.
saagarjha 5y ago
Considering that all build systems must keep a dependency graph internally, it’s surprising how many of them make it incredibly difficult to inspect why things are being rebuilt. I don’t really get it either, I’ve talked to a lot of people and getting feedback on how the build system is doing its task is one of the biggest complaints I see (others are “what is the build system doing” and “why does this have the worst syntax and weird edge cases”) and I mean all the build tools have literally everything they need to show this externally, but basically none expose it in a reasonable manner (if at all). I should be able to go, “oh you rebuilt all of libFoo.a because I accidentally made it depend on autogenerated.cpp”. If you are making a new build system today (and I am sure some of you here are considering it :P) please make this one of your priorities when designing the tool.
- pjc50 5y ago
  Msbuild (as used in Visual Studio and dotnet core) has an amazing implementation of this: https://msbuildlog.com/ https://msbuildlog.com/ Not only can you see what was rebuilt and why, you can see all the input into the rules at all the stages of the build, conveniently hyperlinked into the master rule files provided in the SDK.
- eddieh 5y ago
  GNU Make is pretty good about telling you why things are happening with -d If you can get away with it use -rR to skip builtins, it makes the output of -d much nicer. I usually end up running make with -rRd or -rRp when I want to inspect things. -d tells you why things happen and -p shows you what rules and variables are defined.
  - sideshowb 5y ago
    My bugbear with make (and even the better remake) is finding out why things are not happening. "No rule to make target" is such an uninformative error message. Could it not identify rules which would make the target were they not failing?
- jhasse 5y ago
  Like ninja -d explain ?
  - drothlis 5y ago
    Ninja's "-d explain" is next to useless in a large build: It says what's dirty but it's hard to relate which dirty input caused which output to be re-built. At the beginning of your build you get pages & pages of "these things are dirty" and then it starts the actual build. Also, this "explain" output is printed before ninja has realised that a lot of those things aren't actually dirty (thanks to "restat"). I've been using this in my build system to make the explain output slightly more useful: https://github.com/ninja-build/ninja/pull/2067 https://github.com/ninja-build/ninja/pull/2067 It's just a quick hack that took me half a day (most of which was remembering how to C++ after 8 years...) but I think it makes "-d explain" much more useful. Really, the "explain" implementation could do with an overhaul. I'd love ninja to serialise the build graph to disk (or maybe just the subset of the graph that actually needed rebuilding). This could be augmented with the reason why each rule was run, and possibly even with the command output for each rule. Then the existing ninja tools like "graph", "browse", and "query" could optionally use this graph instead of the "build.ninja" file, so that you can inspect what happened the last time you ran "ninja". Realistically I'll never get around to implementing this, and my pull request above gets me close enough.
jgrahamc 5y ago
You can hack some of this functionality into GNU Make makefiles by modifying $(SHELL). https://www.cmcrossroads.com/article/tracing-rule-execution-gnu-make https://www.cmcrossroads.com/article/tracing-rule-execution-...
- dolmen 5y ago
  Clever hack!
glandium 5y ago
See also: https://github.com/rocky/remake https://github.com/rocky/remake ISTR there was a thread about it here a few months ago.
sydthrowaway 5y ago
Ive tried this before but with a complex build the graph is unwieldy and useless
- kergonath 5y ago
  It’s probably more suited for the OP’s use case (a couple of files causing some re-building) because otherwise yeah, the build tree can be humongous. That said, isn’t there some software to visualise dot files in a more sophisticated way than just generating a picture? It would be handy to be able to collapse/expand sub graphs in a GUI, for example.
  - jcranmer 5y ago
    There's a couple of interactive graph visualization software out there that could be used instead of static dot file generation. (One of the advantages of DOT is that it's a pretty common de facto standard for describing graphs, so you can import it into most graph manipulation libraries and mangle it further using that.)
  - datashaman 5y ago
    Something like this could be handy. (Disclaimer: never used it) https://pypi.org/project/pydot/ https://pypi.org/project/pydot/
  - JNRowe 5y ago
    [Noting that you said GUI, and what I'm about to say isn't] There's a lot more to Graphviz than just dot, as it comes with some really good filtering tools that people often seem to miss. You can achieve an awful lot with just bcomps/ccomps. And, when you're running queries gc is useful to see that you're heading down the right path.
    - kergonath 5y ago
      Yes, that could also be helpful to split a complicated build graph into more manageable pieces. I had a look at the tools available for an unrelated project a while ago; I probably need to see what’s available now. Hopefully the tools have progressed a bit.
indygreg2 5y ago
I once helped maintain a nearly full featured implementation of make in Python (pymake) that grew out of the Firefox project. As much as I would like to support efforts like this, I feel like it is ultimately doomed to suffer from usability limitations because make does not have a static DAG. Rather, the DAG evolves dynamically as make evaluates targets. There are pesky problems like $(call) and $(eval) where you can dynamically inject make expressions into a partially evaluated DAG. And targets can generate files which are later loaded via include directives. Dumping the make database in a machine readable format (for visualization or otherwise) would be incredibly valuable for debugging and could improve understanding. But since many make files have N>>1 "snapshots" of the internal DAG during their execution, there is a really thorny problem around when and which of these "snapshots" to use and how to stitch them together. Many make files aren't "static" enough to support exporting a single, complete, and valuable usable snapshot of their internal DAG. If debugging is the target goal, I think a potentially better approach would be to define a function that takes an output filename and optional list of targets and writes out the point-in-time build graph when that function is called. This way makefile authors could insert probes in the make file (including at the bottom of a make file so the function is called on file load) and capture the state(s) of the build graph exactly when they need to. There could also be a specially named variable holding the names of targets that when pre- or post-evaluated would trigger the dumping of the database. Best of luck to the person doing this work. Debugging make is notoriously hard and any progress in this area will be much welcomed by its many users.
- gjvc 5y ago
  Does pymake handle colons in filenames any better / easier / etc than GNU make? (I have a task where I have a bunch of files to transform and they have HH:MM:DD timestamps embedded in the basename of the file)
- deleted 5y ago
  [deleted]
- loo 5y ago
  This must be why after 45 years make still has no "list targets" option.
eqvinox 5y ago
Wouldn't it be possible/easier to create a new/separate script that converts the `make -p` output to the desired `.dot` graph? (Just to have this not be a thing that needs to pass through the GNU make project.) Also I feel like on most nontrivial projects the graph would be gigantic & to get any usability you'd need something interactive where you can collapse/expand the nodes you're (not) interested in. Don't we have a bunch of JS libraries these days that provide this?
- mhh__ 5y ago
  It might be easier but having tooling build info your... tooling is genuinely useful
vasvir 5y ago
Well there is makefile2graph https://github.com/lindenb/makefile2graph https://github.com/lindenb/makefile2graph. It is even packaged be Debian and it does exactly that. I use it like this from inside the Makefile: make-db.png: make -f $(MAKEFILE) -Bnd | sed -e 's,$(MAKEDIR)/,,g' | make2graph | dot -Tpng -o $@ but with some gymnastics in order to remove very long path names (MAKEDIR) can be used from the shell directly.
- etangent 5y ago
  See also: https://github.com/souring001/makefile-visualizer https://github.com/souring001/makefile-visualizer
EdSchouten 5y ago
Funnily enough, Bazel/Blaze has already supported this for at least a decade. You can just run: bazel query ${some_expression} -output graph | dot -Tpdf > out.pdf https://docs.bazel.build/versions/main/query-how-to.html https://docs.bazel.build/versions/main/query-how-to.html
Ericson2314 5y ago
I would very much like this for https://github.com/NixOS/rfcs/pull/92 https://github.com/NixOS/rfcs/pull/92
LAC-Tech 5y ago
I get the impression that GNU Make is a fairly hard program to update - to the best of my knowledge it still can't handle filenames with spaces in them, for example. So for me it's a lot like a lot of GNU projects - great idea, but a little bit long in the tooth and unlikely to substantially improve at this point.
deleted 5y ago
[deleted]
asdff 5y ago
snakemake has dag output built in
ReleaseCandidat 5y ago
Irix (SGI's) had a make GUI frontend called 'Build Manager' that had a graph view called 'Build Analyzer' of the dependencies. You could even run 'what would happen, if I change this file?' queries on the graph. https://techpubs.jurassic.nl/manuals/0650/developer/Debugger_UG/sgi_html/apa.html https://techpubs.jurassic.nl/manuals/0650/developer/Debugger...
tpoacher 5y ago
This may be a naive comment, but, what's stopping one from simply appending a graphviz command at the end of each recipe? (manually or otherwise)
danny0z 5y ago
xmake can also easily write some lua scripts to dump the build dependency graph https://github.com/xmake-io/xmake https://github.com/xmake-io/xmake
mplanchard 5y ago
I have found the `--debug=v` flag to be super helpful for debugging make targets. The `v` flag generates a lot more output than `--debug` with no arguments. Would also love a graph though

161 points by jonasdn 5y ago

Some clarification needed, > The Android build system works by including all recipes to be built (programs / libraries / etc) using the GNU Make include directive, so that you end up with one giant Makefile that holds all rules for building the platform. Possibly to avoid the problems laid out in the paper Recursive make considered harmful. The old build system (ndk-build) works like this. Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong.

darkamaul 5y ago
> Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong. Soong build files are easy to parse and you can create an Unified Dependency Graph from them. That's what I did in BGraph [0] for security applications. [0] https://github.com/quarkslab/bgraph https://github.com/quarkslab/bgraph
malkia 5y ago
AFAIK, Soong -> Bazel - https://blog.bazel.build/2020/11/12/aosp_migrating_to_bazel.html https://blog.bazel.build/2020/11/12/aosp_migrating_to_bazel.... (2020)

kergonath 5y ago

This would be quite useful. We have a couple of large-ish projects with a couple of hundreds files compiled in parallel and there are snags every now and then (which disappear when re-running make either in parallel or sequential modes). Getting a graph in these cases would be a nice time saver. I’ll try the fork.

saagarjha 5y ago

Considering that all build systems must keep a dependency graph internally, it’s surprising how many of them make it incredibly difficult to inspect why things are being rebuilt. I don’t really get it either, I’ve talked to a lot of people and getting feedback on how the build system is doing its task is one of the biggest complaints I see (others are “what is the build system doing” and “why does this have the worst syntax and weird edge cases”) and I mean all the build tools have literally everything they need to show this externally, but basically none expose it in a reasonable manner (if at all). I should be able to go, “oh you rebuilt all of libFoo.a because I accidentally made it depend on autogenerated.cpp”. If you are making a new build system today (and I am sure some of you here are considering it :P) please make this one of your priorities when designing the tool.

pjc50 5y ago
Msbuild (as used in Visual Studio and dotnet core) has an amazing implementation of this: https://msbuildlog.com/ https://msbuildlog.com/ Not only can you see what was rebuilt and why, you can see all the input into the rules at all the stages of the build, conveniently hyperlinked into the master rule files provided in the SDK.
eddieh 5y ago
GNU Make is pretty good about telling you why things are happening with -d If you can get away with it use -rR to skip builtins, it makes the output of -d much nicer. I usually end up running make with -rRd or -rRp when I want to inspect things. -d tells you why things happen and -p shows you what rules and variables are defined.
- sideshowb 5y ago
  My bugbear with make (and even the better remake) is finding out why things are not happening. "No rule to make target" is such an uninformative error message. Could it not identify rules which would make the target were they not failing?
jhasse 5y ago
Like ninja -d explain ?
- drothlis 5y ago
  Ninja's "-d explain" is next to useless in a large build: It says what's dirty but it's hard to relate which dirty input caused which output to be re-built. At the beginning of your build you get pages & pages of "these things are dirty" and then it starts the actual build. Also, this "explain" output is printed before ninja has realised that a lot of those things aren't actually dirty (thanks to "restat"). I've been using this in my build system to make the explain output slightly more useful: https://github.com/ninja-build/ninja/pull/2067 https://github.com/ninja-build/ninja/pull/2067 It's just a quick hack that took me half a day (most of which was remembering how to C++ after 8 years...) but I think it makes "-d explain" much more useful. Really, the "explain" implementation could do with an overhaul. I'd love ninja to serialise the build graph to disk (or maybe just the subset of the graph that actually needed rebuilding). This could be augmented with the reason why each rule was run, and possibly even with the command output for each rule. Then the existing ninja tools like "graph", "browse", and "query" could optionally use this graph instead of the "build.ninja" file, so that you can inspect what happened the last time you ran "ninja". Realistically I'll never get around to implementing this, and my pull request above gets me close enough.

jgrahamc 5y ago

You can hack some of this functionality into GNU Make makefiles by modifying $(SHELL). https://www.cmcrossroads.com/article/tracing-rule-execution-gnu-make https://www.cmcrossroads.com/article/tracing-rule-execution-...

dolmen 5y ago
Clever hack!

glandium 5y ago

See also: https://github.com/rocky/remake https://github.com/rocky/remake ISTR there was a thread about it here a few months ago.

sydthrowaway 5y ago

Ive tried this before but with a complex build the graph is unwieldy and useless

kergonath 5y ago
It’s probably more suited for the OP’s use case (a couple of files causing some re-building) because otherwise yeah, the build tree can be humongous. That said, isn’t there some software to visualise dot files in a more sophisticated way than just generating a picture? It would be handy to be able to collapse/expand sub graphs in a GUI, for example.
- jcranmer 5y ago
  There's a couple of interactive graph visualization software out there that could be used instead of static dot file generation. (One of the advantages of DOT is that it's a pretty common de facto standard for describing graphs, so you can import it into most graph manipulation libraries and mangle it further using that.)
- datashaman 5y ago
  Something like this could be handy. (Disclaimer: never used it) https://pypi.org/project/pydot/ https://pypi.org/project/pydot/
- JNRowe 5y ago
  [Noting that you said GUI, and what I'm about to say isn't] There's a lot more to Graphviz than just dot, as it comes with some really good filtering tools that people often seem to miss. You can achieve an awful lot with just bcomps/ccomps. And, when you're running queries gc is useful to see that you're heading down the right path.
  - kergonath 5y ago
    Yes, that could also be helpful to split a complicated build graph into more manageable pieces. I had a look at the tools available for an unrelated project a while ago; I probably need to see what’s available now. Hopefully the tools have progressed a bit.

indygreg2 5y ago

I once helped maintain a nearly full featured implementation of make in Python (pymake) that grew out of the Firefox project. As much as I would like to support efforts like this, I feel like it is ultimately doomed to suffer from usability limitations because make does not have a static DAG. Rather, the DAG evolves dynamically as make evaluates targets. There are pesky problems like $(call) and $(eval) where you can dynamically inject make expressions into a partially evaluated DAG. And targets can generate files which are later loaded via include directives. Dumping the make database in a machine readable format (for visualization or otherwise) would be incredibly valuable for debugging and could improve understanding. But since many make files have N>>1 "snapshots" of the internal DAG during their execution, there is a really thorny problem around when and which of these "snapshots" to use and how to stitch them together. Many make files aren't "static" enough to support exporting a single, complete, and valuable usable snapshot of their internal DAG. If debugging is the target goal, I think a potentially better approach would be to define a function that takes an output filename and optional list of targets and writes out the point-in-time build graph when that function is called. This way makefile authors could insert probes in the make file (including at the bottom of a make file so the function is called on file load) and capture the state(s) of the build graph exactly when they need to. There could also be a specially named variable holding the names of targets that when pre- or post-evaluated would trigger the dumping of the database. Best of luck to the person doing this work. Debugging make is notoriously hard and any progress in this area will be much welcomed by its many users.

gjvc 5y ago
Does pymake handle colons in filenames any better / easier / etc than GNU make? (I have a task where I have a bunch of files to transform and they have HH:MM:DD timestamps embedded in the basename of the file)
deleted 5y ago
[deleted]
loo 5y ago
This must be why after 45 years make still has no "list targets" option.

eqvinox 5y ago

Wouldn't it be possible/easier to create a new/separate script that converts the `make -p` output to the desired `.dot` graph? (Just to have this not be a thing that needs to pass through the GNU make project.) Also I feel like on most nontrivial projects the graph would be gigantic & to get any usability you'd need something interactive where you can collapse/expand the nodes you're (not) interested in. Don't we have a bunch of JS libraries these days that provide this?

mhh__ 5y ago
It might be easier but having tooling build info your... tooling is genuinely useful

vasvir 5y ago

Well there is makefile2graph https://github.com/lindenb/makefile2graph https://github.com/lindenb/makefile2graph. It is even packaged be Debian and it does exactly that. I use it like this from inside the Makefile: make-db.png: make -f $(MAKEFILE) -Bnd | sed -e 's,$(MAKEDIR)/,,g' | make2graph | dot -Tpng -o $@ but with some gymnastics in order to remove very long path names (MAKEDIR) can be used from the shell directly.

etangent 5y ago
See also: https://github.com/souring001/makefile-visualizer https://github.com/souring001/makefile-visualizer

EdSchouten 5y ago

Funnily enough, Bazel/Blaze has already supported this for at least a decade. You can just run: bazel query ${some_expression} -output graph | dot -Tpdf > out.pdf https://docs.bazel.build/versions/main/query-how-to.html https://docs.bazel.build/versions/main/query-how-to.html

Ericson2314 5y ago

I would very much like this for https://github.com/NixOS/rfcs/pull/92 https://github.com/NixOS/rfcs/pull/92

LAC-Tech 5y ago

I get the impression that GNU Make is a fairly hard program to update - to the best of my knowledge it still can't handle filenames with spaces in them, for example. So for me it's a lot like a lot of GNU projects - great idea, but a little bit long in the tooth and unlikely to substantially improve at this point.

deleted 5y ago

[deleted]

asdff 5y ago

snakemake has dag output built in

ReleaseCandidat 5y ago

Irix (SGI's) had a make GUI frontend called 'Build Manager' that had a graph view called 'Build Analyzer' of the dependencies. You could even run 'what would happen, if I change this file?' queries on the graph. https://techpubs.jurassic.nl/manuals/0650/developer/Debugger_UG/sgi_html/apa.html https://techpubs.jurassic.nl/manuals/0650/developer/Debugger...

tpoacher 5y ago

This may be a naive comment, but, what's stopping one from simply appending a graphviz command at the end of each recipe? (manually or otherwise)

danny0z 5y ago

xmake can also easily write some lua scripts to dump the build dependency graph https://github.com/xmake-io/xmake https://github.com/xmake-io/xmake

mplanchard 5y ago

I have found the `--debug=v` flag to be super helpful for debugging make targets. The `v` flag generates a lot more output than `--debug` with no arguments. Would also love a graph though