3 ms·

I know make can seem a little baroque, but this is just wrong. 1. Multiple targets for a single recipe: file.a file.b file.c: dep.foo dep.bar ...

I know make can seem a little baroque, but this is just wrong. 1. Multiple targets for a single recipe: file.a file.b file.c: dep.foo dep.bar ... This says that the recipe makes all of file.a, file.b and file.c in one go. 2. Make definitely doesn't randomly delete files. It deletes implicit targets. Make by default knows how to build a lot of standard things like object files for c programs, yacc and bison stuff, etc. These are called implicit targets. These are considered intermediate files to be deleted. You can override the defaults or add your own implicit targets by using pattern matching like this: %.foo: %.bar ... If you want to use pattern matching for non-implicit targets so they don't get deleted, you can do that too: a.foo b.foo c.foo: %.foo: %.bar ... The list before the first colon says which targets the pattern-matched rule applies to and shouldn't contain wildcards. These targets won't get deleted. 3. This seems like a misunderstanding of make's basic role. Make just spawns shells when running a recipe; like bash, it shouldn't need to know how many threads you're using to run an arbitrary command. If you want make to build targets in parallel whenever possible, look at the `-j` option. If you want a certain build recipe to run multi-threaded, use the proper tool for the recipe. 4. Not sure what you mean by crash recovery, but considering the above, I'm pretty sure you might just be fighting make unnecessarily. Honestly, try reading the info manual. It's kind of massive and daunting, but the intro material is really accessible, and taken in pieces, you can easily learn to become friends with this venerable tool.

mnw21cam 9y ago
1. That doesn't do what you think it does. From the manual: "A rule with multiple targets is equivalent to writing many rules, each with one target, and all identical aside from that." It does not mean one rule that creates multiple targets. To achieve that, you need to use wildcards. For some reason, when using wildcards, the syntax is interpreted differently. 2. If I create a rule to create an intermediate file "b" from original file "a", then another rule to create file "c" that I want to keep from "b", but there is an error running the command that creates "c", then make will happily delete the intermediate "b" (which in my case took 27 hours to create) although it knows the final "c" wasn't created properly. This means that when I rerun make (having fixed the problem), that 27 hour process needs to be run again, which is a waste of my time. 3. I want to say "make -j 64" on my 64-thread server, and not have 64 64-thread processes start. But I also do want 64 single-threaded processes to run when possible. 4. By crash recovery, I mean that by default a process will start creating the target file. If someone yanks the power, that target file will be present, with a recent modified time, but incomplete. Make will assume the file was created fine, so when I rerun make it will try to perform the next step, which may take 10 hours to fail. I want make to notice that the command did not successfully complete, and restart it from scratch.
- chubot 9y ago
  I agree with all your criticisms, except I'm a bit confused about #3. Are you saying you're using Make with multi-threaded build actions? As far as I know, most compilers are single-threaded, so this isn't much of an issue in practice. But I'm curious where you've encountered this problem.
  - mnw21cam 9y ago
    No, I was using make to process large files for bioinformatics. So, think 60GB (compressed) of sequencing data from a whole genome sequencing run, which comes as a set of ~800,000,000 individually sequenced short stretches of DNA in two files. A multi-threaded process converts that into a file containing the sequences and where they align in the human genome, and takes about a day. Once that job has been finished, other jobs can be kicked off to use the produced data. Overall, the build process is a DAG with several hundred individual jobs, and performing that in a make-like system helps it to be managed effectively. Just not make itself.
- fusiongyro 9y ago
  For #2, you can mark intermediates worth keeping with .PRECIOUS. https://www.gnu.org/software/make/manual/html_node/Special-Targets.html https://www.gnu.org/software/make/manual/html_node/Special-T... For #3, I think you may be misreading ps; on Linux, ps will show you threads as if they are processes when they are not.
  - mnw21cam 9y ago
    For #2, .PRECIOUS doesn't help me. From the make manual: "Also, if the target is an intermediate file, it will not be deleted after it is no longer needed, as is normally done." This means that my intermediate files will never be deleted by make, even when everything that is built from them has been completed. For #3, no I think I know how to read ps. I don't want 64 64-thread processes running on my 64-thread server, because that is hell for an OS scheduler, and makes things run slower, not faster.
    - 9y ago
chubot 9y ago
#1 -- No, this is the "obvious but wrong" solution. It doesn't work for parallel builds. https://www.cmcrossroads.com/article/rules-multiple-outputs-gnu-make https://www.cmcrossroads.com/article/rules-multiple-outputs-... All his criticisms are correct except maybe #3 which I don't understand. Another problem I've found is that Make doesn't consider the absence of a prequisite to mean the target is out of date. So if foo.html depends on foo.intermediate, and then you delete foo.intermediate, then "make foo.html", foo.html will be considered up to date. I guess this is part of the odd feature where Make deletes intermediaries, but even if you have .SECONDARY on, which I do, it still behaves this way. The bottom line is that it's extraordinarily easy to write incorrect Makefiles -- either underspecifying or overspecifying dependencies -- and it's very difficult to debug those problems. My Makefile is still full of bugs, so I "make clean" when something goes wrong. One thing that would go a long way is if it had a shorthand for referring to the prequisites in commands, like $P1 $P2 $P3, and if it actually enforced that you use those in the command lines! I don't want to create variables for every single file, and when I rename files, rules can grow invisible bugs easily. Some details here: http://www.oilshell.org/blog/2017/05/31.html http://www.oilshell.org/blog/2017/05/31.html

mnw21cam 9y ago

1. That doesn't do what you think it does. From the manual: "A rule with multiple targets is equivalent to writing many rules, each with one target, and all identical aside from that." It does not mean one rule that creates multiple targets. To achieve that, you need to use wildcards. For some reason, when using wildcards, the syntax is interpreted differently. 2. If I create a rule to create an intermediate file "b" from original file "a", then another rule to create file "c" that I want to keep from "b", but there is an error running the command that creates "c", then make will happily delete the intermediate "b" (which in my case took 27 hours to create) although it knows the final "c" wasn't created properly. This means that when I rerun make (having fixed the problem), that 27 hour process needs to be run again, which is a waste of my time. 3. I want to say "make -j 64" on my 64-thread server, and not have 64 64-thread processes start. But I also do want 64 single-threaded processes to run when possible. 4. By crash recovery, I mean that by default a process will start creating the target file. If someone yanks the power, that target file will be present, with a recent modified time, but incomplete. Make will assume the file was created fine, so when I rerun make it will try to perform the next step, which may take 10 hours to fail. I want make to notice that the command did not successfully complete, and restart it from scratch.

chubot 9y ago
I agree with all your criticisms, except I'm a bit confused about #3. Are you saying you're using Make with multi-threaded build actions? As far as I know, most compilers are single-threaded, so this isn't much of an issue in practice. But I'm curious where you've encountered this problem.
- mnw21cam 9y ago
  No, I was using make to process large files for bioinformatics. So, think 60GB (compressed) of sequencing data from a whole genome sequencing run, which comes as a set of ~800,000,000 individually sequenced short stretches of DNA in two files. A multi-threaded process converts that into a file containing the sequences and where they align in the human genome, and takes about a day. Once that job has been finished, other jobs can be kicked off to use the produced data. Overall, the build process is a DAG with several hundred individual jobs, and performing that in a make-like system helps it to be managed effectively. Just not make itself.
fusiongyro 9y ago
For #2, you can mark intermediates worth keeping with .PRECIOUS. https://www.gnu.org/software/make/manual/html_node/Special-Targets.html https://www.gnu.org/software/make/manual/html_node/Special-T... For #3, I think you may be misreading ps; on Linux, ps will show you threads as if they are processes when they are not.
- mnw21cam 9y ago
  For #2, .PRECIOUS doesn't help me. From the make manual: "Also, if the target is an intermediate file, it will not be deleted after it is no longer needed, as is normally done." This means that my intermediate files will never be deleted by make, even when everything that is built from them has been completed. For #3, no I think I know how to read ps. I don't want 64 64-thread processes running on my 64-thread server, because that is hell for an OS scheduler, and makes things run slower, not faster.
  - 9y ago

chubot 9y ago

#1 -- No, this is the "obvious but wrong" solution. It doesn't work for parallel builds. https://www.cmcrossroads.com/article/rules-multiple-outputs-gnu-make https://www.cmcrossroads.com/article/rules-multiple-outputs-... All his criticisms are correct except maybe #3 which I don't understand. Another problem I've found is that Make doesn't consider the absence of a prequisite to mean the target is out of date. So if foo.html depends on foo.intermediate, and then you delete foo.intermediate, then "make foo.html", foo.html will be considered up to date. I guess this is part of the odd feature where Make deletes intermediaries, but even if you have .SECONDARY on, which I do, it still behaves this way. The bottom line is that it's extraordinarily easy to write incorrect Makefiles -- either underspecifying or overspecifying dependencies -- and it's very difficult to debug those problems. My Makefile is still full of bugs, so I "make clean" when something goes wrong. One thing that would go a long way is if it had a shorthand for referring to the prequisites in commands, like $P1 $P2 $P3, and if it actually enforced that you use those in the command lines! I don't want to create variables for every single file, and when I rename files, rules can grow invisible bugs easily. Some details here: http://www.oilshell.org/blog/2017/05/31.html http://www.oilshell.org/blog/2017/05/31.html