3 ms·

I don't know a lot of makefiles, but based on what was written in the article, I was just trying to design a makefile which enforces goal ordering using the "co

by xg15 4y ago

I don't know a lot of makefiles, but based on what was written in the article, I was just trying to design a makefile which enforces goal ordering using the "correct" way, i.e dependencies. Such a makefile should naturally be unaffected by --shuffle and by parallel execution. Do I miss something or is this impossible short of duplicating the goals? E.g., suppose you have goals a, b and c, which are independent and you want to define another goal d which runs first a, then b then c. My first idea was to define additional goals to capture the dependency structure, like this: d: x2 c x2: x1 b x1: a But then nothing is stopping the shuffler from running c, b, a, x1, x2, d. The only solution I see would be to copy the shell commands of b and c into new goals: d: xc xc: xb; <shell commands of c> xb: a; <shell commands of b> Which seems really unsatisfying. Is there a better way?

lumb63 4y ago
If a, b, and c are independent and d depends on them all, you could declare: d: a b c With shuffle, a, b, and c could be made in any order, but you are guaranteed d would be after all of them. This helps to enforce that a, b, and c are actually independent. If they are not, there is a chance your build will fail. This can happen sometimes when rules get complex and there become hidden dependencies between rules. For example, maybe one target, which always happens to run first, creates a directory that other targets rely on, but those targets would not create that directory if they were run first. That is the impetus behind this new flag. It would (maybe) expose that there is a hidden dependency on that directory, and it could be made into its own dependency for a, b, and c to depend on, so that it doesn’t matter which runs first.
delusional 4y ago
I think you're correct, it's impossible to instruct make to order your dependencies since dependencies may be built in parallel. I'm having trouble thinking of a case where it would be useful to do so.
Karellen 4y ago
Why is it that you want d to run a then b then c, if a, b and c are independent goals? It might be that `make` is the wrong tool for the job here, but I'm not sure what job you're trying to accomplish. You might be having an XY problem. https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem
- xg15 4y ago
  In theory yes. But a lot of people seem to think they need this and (wrongly) assume they can archieve it by simply writing the goals in order. Otherwise OP wouldn't be able to break so many builds with the --shuffle option. I could see it make sense for some "generic" goals which have different dependencies depending on the context they are called. E.g. a "package" goal requiring that something is there to package, but what is there might depend on the top-level goal. But I agree, this is a bit of a misuse of goals - and definitely the direction that ultimately leads to some makefile from hell when this is invariably overdone.
  - saurik 4y ago
    There is no such thing as "the" top-level goal, so that use case doesn't work: if someone does a parallel build of multiple "top-level" goals the package goal will still need to do something reasonable. To make this work you need multiple package goals, maybe using a pattern goal, and then the package--which sounds like the thing he user was actually trying to get--is going to be an ultimate goal, not whatever you previously thought was the "top-level" one. Regardless: I haven't tried it yet, but I saw in a changelog that a recent version of make added .WAIT, which sounded like it does what you want.
  - Karellen 4y ago
    > But a lot of people seem to think they need this and (wrongly) assume they can archieve it by simply writing the goals in order. Do they? > Otherwise OP wouldn't be able to break so many builds with the --shuffle option. That's not what I took away from OPs post. My reading is that a lot of people have goals that they think are independent, but contain hidden dependencies that they'd just not considered. Or, they have goals which have dependencies that were introduced after the makefile was written, and the makefile hasn't caught up yet. i.e. they don't want "build a then b then c", they want "build a and b and c together if that's at all possible (because I have 48 cores and only a single-threaded compiler) but not if it isn't" - and they just haven't correctly specified what should be possible. Having thought a bit more, you could do your original request with the following stanza: d: make a make b make c If that's what they wanted, it's not that hard. Or they could just write a shell script to do a then b then c, instead of using make.
teawrecks 4y ago
If they need to run in order then they have a dependency on each other and you have to express that.
smallnamespace 4y ago
Other sibling comments express that you can push the dependency graph into the targets, but if you really do need a-d to be built in order then it seems more straightforward to just execute them all as a single target: FORCE: a b c d: FORCE run-a && run-b && run-c && run-d
- jgrahamc 4y ago
  That's not a "single target". That's four targets and is equivalent to: FORCE: a: FORCE run-a && run-b && run-c && run-d b: FORCE run-a && run-b && run-c && run-d c: FORCE run-a && run-b && run-c && run-d d: FORCE run-a && run-b && run-c && run-d
  - smallnamespace 4y ago
    Thanks for the catch, to get the desired behavior you can use 'Grouped Targets'[1] with the following syntax: a b c d &: FORCE ... Note that this is a GNU make-ism, not sure what BSD make does. [1] https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html https://www.gnu.org/software/make/manual/html_node/Multiple-...
    - jgrahamc 4y ago
      Yeah, but that's such a recent addition I tend to avoid it.
GuB-42 4y ago
The way I do it is by calling make recursively. THIS_FILE := '$(lastword $(MAKEFILE_LIST))' d: $(MAKE) -f $(THIS_FILE) a $(MAKE) -f $(THIS_FILE) b $(MAKE) -f $(THIS_FILE) c
- asveikau 4y ago
  This hides the work of the subprocess from the parent. So if a, b, or c depend on other objects, make would need to stat them to discover they are built, maybe make -j could lead to them being built in parallel creating a race condition. It's better to leave the whole dependency tree in the same makefile by not having a make subprocess, then let make do its dependency magic correctly with full information, enabling -j to get a parallel build.
  - GuB-42 4y ago
    Yes, I see what you mean. In practice, I have only used this trick for "clean and build" rules, which mean that d will never depend on the dependencies of a, b, and c, and I take care not to invoke several conflicting rules as the same time on the command line. Maybe make is able to do some magic to avoid this problem since it has some some kind of process management, at least for dealing with "-j" properly in recursive Makefiles. I doubt it though.

by xg15 4y ago

lumb63 4y ago

If a, b, and c are independent and d depends on them all, you could declare: d: a b c With shuffle, a, b, and c could be made in any order, but you are guaranteed d would be after all of them. This helps to enforce that a, b, and c are actually independent. If they are not, there is a chance your build will fail. This can happen sometimes when rules get complex and there become hidden dependencies between rules. For example, maybe one target, which always happens to run first, creates a directory that other targets rely on, but those targets would not create that directory if they were run first. That is the impetus behind this new flag. It would (maybe) expose that there is a hidden dependency on that directory, and it could be made into its own dependency for a, b, and c to depend on, so that it doesn’t matter which runs first.

delusional 4y ago

I think you're correct, it's impossible to instruct make to order your dependencies since dependencies may be built in parallel. I'm having trouble thinking of a case where it would be useful to do so.

Karellen 4y ago

Why is it that you want d to run a then b then c, if a, b and c are independent goals? It might be that `make` is the wrong tool for the job here, but I'm not sure what job you're trying to accomplish. You might be having an XY problem. https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem

xg15 4y ago
In theory yes. But a lot of people seem to think they need this and (wrongly) assume they can archieve it by simply writing the goals in order. Otherwise OP wouldn't be able to break so many builds with the --shuffle option. I could see it make sense for some "generic" goals which have different dependencies depending on the context they are called. E.g. a "package" goal requiring that something is there to package, but what is there might depend on the top-level goal. But I agree, this is a bit of a misuse of goals - and definitely the direction that ultimately leads to some makefile from hell when this is invariably overdone.
- saurik 4y ago
  There is no such thing as "the" top-level goal, so that use case doesn't work: if someone does a parallel build of multiple "top-level" goals the package goal will still need to do something reasonable. To make this work you need multiple package goals, maybe using a pattern goal, and then the package--which sounds like the thing he user was actually trying to get--is going to be an ultimate goal, not whatever you previously thought was the "top-level" one. Regardless: I haven't tried it yet, but I saw in a changelog that a recent version of make added .WAIT, which sounded like it does what you want.
- Karellen 4y ago
  > But a lot of people seem to think they need this and (wrongly) assume they can archieve it by simply writing the goals in order. Do they? > Otherwise OP wouldn't be able to break so many builds with the --shuffle option. That's not what I took away from OPs post. My reading is that a lot of people have goals that they think are independent, but contain hidden dependencies that they'd just not considered. Or, they have goals which have dependencies that were introduced after the makefile was written, and the makefile hasn't caught up yet. i.e. they don't want "build a then b then c", they want "build a and b and c together if that's at all possible (because I have 48 cores and only a single-threaded compiler) but not if it isn't" - and they just haven't correctly specified what should be possible. Having thought a bit more, you could do your original request with the following stanza: d: make a make b make c If that's what they wanted, it's not that hard. Or they could just write a shell script to do a then b then c, instead of using make.

teawrecks 4y ago

If they need to run in order then they have a dependency on each other and you have to express that.

smallnamespace 4y ago

Other sibling comments express that you can push the dependency graph into the targets, but if you really do need a-d to be built in order then it seems more straightforward to just execute them all as a single target: FORCE: a b c d: FORCE run-a && run-b && run-c && run-d

jgrahamc 4y ago
That's not a "single target". That's four targets and is equivalent to: FORCE: a: FORCE run-a && run-b && run-c && run-d b: FORCE run-a && run-b && run-c && run-d c: FORCE run-a && run-b && run-c && run-d d: FORCE run-a && run-b && run-c && run-d
- smallnamespace 4y ago
  Thanks for the catch, to get the desired behavior you can use 'Grouped Targets'[1] with the following syntax: a b c d &: FORCE ... Note that this is a GNU make-ism, not sure what BSD make does. [1] https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html https://www.gnu.org/software/make/manual/html_node/Multiple-...
  - jgrahamc 4y ago
    Yeah, but that's such a recent addition I tend to avoid it.

GuB-42 4y ago

The way I do it is by calling make recursively. THIS_FILE := '$(lastword $(MAKEFILE_LIST))' d: $(MAKE) -f $(THIS_FILE) a $(MAKE) -f $(THIS_FILE) b $(MAKE) -f $(THIS_FILE) c

asveikau 4y ago
This hides the work of the subprocess from the parent. So if a, b, or c depend on other objects, make would need to stat them to discover they are built, maybe make -j could lead to them being built in parallel creating a race condition. It's better to leave the whole dependency tree in the same makefile by not having a make subprocess, then let make do its dependency magic correctly with full information, enabling -j to get a parallel build.
- GuB-42 4y ago
  Yes, I see what you mean. In practice, I have only used this trick for "clean and build" rules, which mean that d will never depend on the dependencies of a, b, and c, and I take care not to invoke several conflicting rules as the same time on the command line. Maybe make is able to do some magic to avoid this problem since it has some some kind of process management, at least for dealing with "-j" properly in recursive Makefiles. I doubt it though.