3 ms·

If your project sports auto-generated code, or any sort of build targets that depend on inputs only known after generation -- Make simply cannot handle this. Th

by Peaker 14y ago

If your project sports auto-generated code, or any sort of build targets that depend on inputs only known after generation -- Make simply cannot handle this. There are ugly workarounds but they don't work well. Make is: * Slow (e.g: when compared with tup[1]) * Very easy to get it wrong (under-specify dependencies), with cryptic bugs (or over-building) as a result. Pretty difficult to get it right (e.g: doing proper #include scanning, especially with the above-mentioned generated code). * Terrible scripting language (mentioned by many others) * Lacks useful features of other build systems (e.g: tup, or shake) of useful profile reports and queries about the build process. It is just an antiquated, poor tool that ought to be replaced by betters that already exist, whenever possible. [1]: http://gittup.org/tup/ http://gittup.org/tup/

chubot 14y ago
Have you used tup? I have encountered on the web, and it looks very well done. But I never hear about anyone using it. Is it just under-publicized? People sometimes talk about waf as an alternative. I took a look at waf awhile aback and I felt disappointed, I forget why. I think node.js switched off waf for some reason I don't recall; either that or they didn't like it.
- Peaker 14y ago
  I used tup, and it has its shortcomings. I would never prefer Make to tup though, even with its shortcomings. I am now working with Shake, and it seems to be the nicest I've used yet. It is not as robust as tup at verifying there are no under-specified dependencies, nor at detecting what needs to be rebuilt when a build script changes. But it is much more flexible (you get to write a build script with the full power of Haskell), it generates much nicer reports as a result, and it has some other interesting features that tup lacks. Unlike tup, you don't need to specify all the dependencies statically before building anything. Since tup has some useful features that Shake lacks, I don't think there's a clear winner here. But either one is far preferable to Make in every setting.
  - chubot 14y ago
    What are the shortcomings of tup? That you have to specify dependencies statically?
    - Peaker 14y ago
      Yeah, though it is not that horrible since tup allows you to over-specify dependencies with very minor ill-effects (less parallelism, but no over-building). My list of short-comings besides that are: * No "run" command on Windows, means if you want portable tup you're stuck with its primitive scripting language (or reverting to Make-style hacks like ugly multi-phase build that generates the Tupfiles) * Has an arbitrary restriction on rule line size. So if you do use "run", and have large targets that have tons of dependencies, you may hit a dreaded: "Line too long" error. * Has some trouble because of its fuse-based system to capture dependencies. Commands are exposed to weird paths in ~/.tup/mnt/ and such. The abstraction sometimes leaks. All in all, these short-comings are very minor compared to Make's huge ones :)
      - quotemstr 14y ago
        > Yeah, though it is not that horrible since tup allows you to over-specify dependencies with very minor ill-effects (less parallelism, but no over-building). Make has exactly the same behavior with respect to additional dependencies. > All in all, these short-comings are very minor compared to Make's huge ones Like what?
        Peaker 14y ago
        > Make has exactly the same behavior with respect to additional dependencies No it doesn't. If I overspecify that foo.o depends on foo.h, and I change foo.h, Make will rebuild foo.o even if it doesn't actually depend on foo.h. Tup won't. > Like what? Like incorrect builds, not supporting auto-generated code with dependency scanning, not scaling to large projects (slow builds, as demo'd by the benchmark page), lack of any kind of reports about the build, no guarantees at all about any output, and more.
      - 14y ago
niggler 14y ago
"or any sort of build targets that depend on inputs only known after generation" Have you tried the gnu make SECONDEXPANSION technique?
- Peaker 14y ago
  Adding a single extra phase is not enough, because you need an arbitrary number of phases to recursively build&scan all dependencies.
  - quotemstr 14y ago
    Can you provide a concrete example of a problem GNU make's secondary expansion feature cannot solve?
    - Peaker 14y ago
      Auto-generate script that generates header0.h with #include to header1.h, and header1.h with #include to header2.h, up to N=10. Of course, we don't know before generating the headers exactly what they'll need to #include. And N is determined solely by the existence of the #include, i.e: generation of header10.h does not #include header11.h and that should stop the build.
      - niggler 14y ago
        you can use most C compilers to autogenerate the dependencies in make format (see http://hastebin.com/kufajaqeso.sh http://hastebin.com/kufajaqeso.sh for a sample makefile and script to generate the relevant code -- there's a commented line in header10.h that you should remove to prove to yourself that it indeed does the right thing)
        Peaker 14y ago
        Your solution isn't a solution, because it is not actually a Makefile. Your shell script and Makefile both need to be run, in the correct order, by a meta build-system. It won't re-generate just the right files when things change. Unless you always regenerate all the code, and that's gonna be very wasteful. It won't parallelize things as possible. The code generation can be parallelized with parts of the build that don't depend on it. It won't rescan just the right files (it will do more work than necessary). Compare that with a real build system, such as tup or shake, that gets all of these properties right.
gyepi 14y ago
> If your project sports auto-generated code, or any sort of > build targets that depend on inputs only known after > generation -- Make simply cannot handle this. > There are ugly workarounds but they don't work well. Hmm. I write a lot of generated code and have not found this to be true at all. Many of my projects have multiple levels of code generation that and make handles them just fine. The key is to play to make's strengths and define pattern rules so you can hook into rules engine. Nothing like typing make and having it know to run a script to generate a text file that is further processed by another tool to generate more files that eventually get compiled to produce what you want.
- Peaker 14y ago
  Can you should an example Makefile that does this correctly? How do you get make to do a multi-phase build, that generates code, then scans it to add to the dependency tree, then generates more, and so forth?

by Peaker 14y ago

chubot 14y ago

Have you used tup? I have encountered on the web, and it looks very well done. But I never hear about anyone using it. Is it just under-publicized? People sometimes talk about waf as an alternative. I took a look at waf awhile aback and I felt disappointed, I forget why. I think node.js switched off waf for some reason I don't recall; either that or they didn't like it.

Peaker 14y ago
I used tup, and it has its shortcomings. I would never prefer Make to tup though, even with its shortcomings. I am now working with Shake, and it seems to be the nicest I've used yet. It is not as robust as tup at verifying there are no under-specified dependencies, nor at detecting what needs to be rebuilt when a build script changes. But it is much more flexible (you get to write a build script with the full power of Haskell), it generates much nicer reports as a result, and it has some other interesting features that tup lacks. Unlike tup, you don't need to specify all the dependencies statically before building anything. Since tup has some useful features that Shake lacks, I don't think there's a clear winner here. But either one is far preferable to Make in every setting.
- chubot 14y ago
  What are the shortcomings of tup? That you have to specify dependencies statically?
  - Peaker 14y ago
    Yeah, though it is not that horrible since tup allows you to over-specify dependencies with very minor ill-effects (less parallelism, but no over-building). My list of short-comings besides that are: * No "run" command on Windows, means if you want portable tup you're stuck with its primitive scripting language (or reverting to Make-style hacks like ugly multi-phase build that generates the Tupfiles) * Has an arbitrary restriction on rule line size. So if you do use "run", and have large targets that have tons of dependencies, you may hit a dreaded: "Line too long" error. * Has some trouble because of its fuse-based system to capture dependencies. Commands are exposed to weird paths in ~/.tup/mnt/ and such. The abstraction sometimes leaks. All in all, these short-comings are very minor compared to Make's huge ones :)
    - quotemstr 14y ago
      > Yeah, though it is not that horrible since tup allows you to over-specify dependencies with very minor ill-effects (less parallelism, but no over-building). Make has exactly the same behavior with respect to additional dependencies. > All in all, these short-comings are very minor compared to Make's huge ones Like what?
      - Peaker 14y ago
        > Make has exactly the same behavior with respect to additional dependencies No it doesn't. If I overspecify that foo.o depends on foo.h, and I change foo.h, Make will rebuild foo.o even if it doesn't actually depend on foo.h. Tup won't. > Like what? Like incorrect builds, not supporting auto-generated code with dependency scanning, not scaling to large projects (slow builds, as demo'd by the benchmark page), lack of any kind of reports about the build, no guarantees at all about any output, and more.
    - 14y ago

niggler 14y ago

"or any sort of build targets that depend on inputs only known after generation" Have you tried the gnu make SECONDEXPANSION technique?

Peaker 14y ago
Adding a single extra phase is not enough, because you need an arbitrary number of phases to recursively build&scan all dependencies.
- quotemstr 14y ago
  Can you provide a concrete example of a problem GNU make's secondary expansion feature cannot solve?
  - Peaker 14y ago
    Auto-generate script that generates header0.h with #include to header1.h, and header1.h with #include to header2.h, up to N=10. Of course, we don't know before generating the headers exactly what they'll need to #include. And N is determined solely by the existence of the #include, i.e: generation of header10.h does not #include header11.h and that should stop the build.
    - niggler 14y ago
      you can use most C compilers to autogenerate the dependencies in make format (see http://hastebin.com/kufajaqeso.sh http://hastebin.com/kufajaqeso.sh for a sample makefile and script to generate the relevant code -- there's a commented line in header10.h that you should remove to prove to yourself that it indeed does the right thing)
      - Peaker 14y ago
        Your solution isn't a solution, because it is not actually a Makefile. Your shell script and Makefile both need to be run, in the correct order, by a meta build-system. It won't re-generate just the right files when things change. Unless you always regenerate all the code, and that's gonna be very wasteful. It won't parallelize things as possible. The code generation can be parallelized with parts of the build that don't depend on it. It won't rescan just the right files (it will do more work than necessary). Compare that with a real build system, such as tup or shake, that gets all of these properties right.

gyepi 14y ago

> If your project sports auto-generated code, or any sort of > build targets that depend on inputs only known after > generation -- Make simply cannot handle this. > There are ugly workarounds but they don't work well. Hmm. I write a lot of generated code and have not found this to be true at all. Many of my projects have multiple levels of code generation that and make handles them just fine. The key is to play to make's strengths and define pattern rules so you can hook into rules engine. Nothing like typing make and having it know to run a script to generate a text file that is further processed by another tool to generate more files that eventually get compiled to produce what you want.

Peaker 14y ago
Can you should an example Makefile that does this correctly? How do you get make to do a multi-phase build, that generates code, then scans it to add to the dependency tree, then generates more, and so forth?