6 ms·

2017 (when this is from) seems to me that it was around the time there was this weird resurgence of people claiming that “modern makefiles” (aka. carefully hand

2017 (when this is from) seems to me that it was around the time there was this weird resurgence of people claiming that “modern makefiles” (aka. carefully handcrafted makefiles) were fine. …but, it was a fad, and it faded, because make is not fine, in any non-trivial incarnation. Yep, sure. A little script for your projects that only you care about and runs on your machine… go for. I’m fond of make for trivial purposes, it’s easy to remember, it’s widely installed. …but, make doesn’t scale; the only way to scale make is to generate your makefiles programmatically; once you do this, your makefiles are no longer makefiles, they’re arcane generated non-portable custom build scripts. Various tools do this; autotools, cmake, etc. Sure, those are tools which are can use if you want, but the elegance and joy of using make vanishes once you use them; you are no longer using make, you are now using a different tool. …so, I have mixed feelings about posts like this. On the one hand, that’s cool; but on the other, you’re really wasting your time learning this stuff because, realistically, you’ll never use it. By the time you need it, you’ll be using a meta tool that generates human-unreadable Makefiles for you. Related, extensive thread: https://news.ycombinator.com/item?id=5275313 https://news.ycombinator.com/item?id=5275313

sjaak 4y ago
> …but, make doesn’t scale; What about these projects? https://github.com/redis/redis/blob/unstable/src/Makefile https://github.com/redis/redis/blob/unstable/src/Makefile https://github.com/torvalds/linux/blob/master/Makefile https://github.com/torvalds/linux/blob/master/Makefile
- wokwokwok 4y ago
  Yes, I can also write a big Makefile to hand compile my project; I can also compile it using shell script that explicitly executes the build steps. …but I dont, not because it’s impossible to do but because the effort in maintaining those is high. Come on, are you serious? Hand crafted makefiles? Have you actually tried to maintain one of those? That redis makefile has a explicitly coded set of object files. Can you imagine trying to maintain that for opencv? I’m just too damn lazy to waste my time maintaining that kind of stuff; it’s a pointless, thankless task… but hey, I’ll give it to you; there are people out there who prefer to do it that way. I posit, they spend more time maintaining their build system then they would prefer, but they keep make for ideological reasons, not practical ones.
- p_l 4y ago
  Are you aware that the first step in building linux kernel is literally bootstrapping a custom build system that just happens to generate makefiles (if not just "mostly make compatible" files - I got crosseyed recently trying to find if kbuild didn't try to run them as well) Linux kernel provides "makefile" as interface for you, but it's mostly compiled from a custom KBuild system.
  - cesarb 4y ago
    AFAIK, the makefile fragment generated by kconfig is nothing more than a set of CONFIG_FOO=y (or =m) statements; everything else is done directly in the makefiles, which have declarations similar to "obj-$(CONFIG_FOO) += foo.o" which become obj-y or obj-m (and so on) depending on the value set by that makefile fragment generated by kconfig. The set of object files to build can then be found in "$(obj-y)" (and so on), which is used as a dependency in a makefile rule to link everything together. The same kconfig also creates a C header file with #define rules for these same CONFIG_ symbols, which is passed to the compiler through a "-include" argument. It's not "mostly compiled from a custom KBuild system", IIRC kbuild is the name given to the whole system, including the makefile rules and macros which do the actual compilation.
    - p_l 4y ago
      Decided to go into the source again, and yes, it's mostly implemented in Makefiles, but they are wildly different from the kind of makefile discussed in the article, with considerable amount of machinery in macros and external C programs (whole "features" thing) to the point that individual makefiles in subdird probably wouldn't parse right if not executed through kbuild wrapper
Joker_vD 4y ago
Honestly, I never quite understood the point of generating the makefiles: if you have a tool that knows where the inputs are, where the outputs should be, and tracks the dependencies changing from build to build (something that "make" famously struggles to do), then surely that tool can also run whatever compilers/linkers/etc. are needed? I guess you can commit the generated makefile into your repo so that those who want to build your project from sources don't need your custom build tool installed, only make...
- tuyiown 4y ago
  Historically, Makefiles generation is merely a cache of platform introspection on systems that were orders of magnitudes slower than what we have now. I suspect that any other approach based on Makefile only would have implied higher memory consumption and/or implied slow rotating disk I/O.
  - account42 4y ago
    Actually on my 64-thread Threadripper that cache is more important than ever as the make step parallelizes nicely but the configure step usually is 100% serialized.
- rossmohax 4y ago
  There are many nuances. How to control parallelism, how to present output, checking freshness, etc. It is much easier to generate Ninja files and let Ninja do all the boring work.
  - zelphirkalt 4y ago
    What are the dependencies of Ninja? (I seem to remember it being a Python tool?)
    - flohofwoe 4y ago
      On macOS at least, Ninja is just a 362 kByte native executable without dependencies except dynamically linking with the system-provided libc++ and libSystem DLLs (you're probably thinking of Meson?)
matheusmoreira 4y ago
Like many others, I too rediscovered the power of pure GNU Make years ago. Turns out it has some rather Lisp-like metaprogramming capabilities. It's extremely hard to do even basic processing but with enough effort some cool things can happen. I'm somewhat proud of my implementation of a simple version of the find utility as a pure GNU Make recursive function. This rediscovery essentially derailed one of my side projects as I started writing a pure GNU Make build system and function library like this one: https://gmsl.sourceforge.io/ https://gmsl.sourceforge.io/ https://www.cmcrossroads.com/article/gnu-make-standard-library https://www.cmcrossroads.com/article/gnu-make-standard-libra... I even asked this author on Twitter and Stack Overflow about its implementation. The end result was reinvention of code generation tools because that's what metaprogramming is. I didn't understand what autotools was doing before I tried doing it myself. My objective was to eliminate makefile regeneration whenever project structure changed, I ended up regenerating code 100% of the time and evaluating the resulting code inside the makefile itself. If we must metaprogram, it's better to do it in a proper programming language. At least it didn't have any dependencies other than GNU Make.
- rasterdog 4y ago
  "Like many others, I too rediscovered the power of pure GNU Make" ... wait a sec, GNU make is not "pure!" I like it but I recognize that it has features that a more portable make would not have.
  - matheusmoreira 4y ago
    Perhaps pure was not the right word. By pure GNU Make I meant writing makefiles myself without higher level makefile generation tools.
- jgrahamc 4y ago
  Hello! That’s me. Happy to answer further questions…
  - matheusmoreira 4y ago
    Hello! Huge fan of your work and I've also read your book. Are you still developing GMSL? I wonder if you're open to contributions. My find function has been the single most useful thing to come out of my side project. The first thing I do in my makefiles is get a list of all source files and it's always been annoying because I like organizing my project in trees which requires recursive file system walks. I managed to write a pure GNU Make function that simplifies that to: sources := $(call find,src,file?) The find function recursively walks the src directory and returns a list of all paths satisfying the file? predicate function. I think it would be a nice addition to GMSL. In my project, I doubt it will ever help anyone.
nocman 4y ago
You offer plenty of criticism for using and/or generating makefiles, but I can't find any place in the discussion where you suggest an alternative. So what is your solution for building large projects?

sjaak 4y ago

> …but, make doesn’t scale; What about these projects? https://github.com/redis/redis/blob/unstable/src/Makefile https://github.com/redis/redis/blob/unstable/src/Makefile https://github.com/torvalds/linux/blob/master/Makefile https://github.com/torvalds/linux/blob/master/Makefile

wokwokwok 4y ago
Yes, I can also write a big Makefile to hand compile my project; I can also compile it using shell script that explicitly executes the build steps. …but I dont, not because it’s impossible to do but because the effort in maintaining those is high. Come on, are you serious? Hand crafted makefiles? Have you actually tried to maintain one of those? That redis makefile has a explicitly coded set of object files. Can you imagine trying to maintain that for opencv? I’m just too damn lazy to waste my time maintaining that kind of stuff; it’s a pointless, thankless task… but hey, I’ll give it to you; there are people out there who prefer to do it that way. I posit, they spend more time maintaining their build system then they would prefer, but they keep make for ideological reasons, not practical ones.
p_l 4y ago
Are you aware that the first step in building linux kernel is literally bootstrapping a custom build system that just happens to generate makefiles (if not just "mostly make compatible" files - I got crosseyed recently trying to find if kbuild didn't try to run them as well) Linux kernel provides "makefile" as interface for you, but it's mostly compiled from a custom KBuild system.
- cesarb 4y ago
  AFAIK, the makefile fragment generated by kconfig is nothing more than a set of CONFIG_FOO=y (or =m) statements; everything else is done directly in the makefiles, which have declarations similar to "obj-$(CONFIG_FOO) += foo.o" which become obj-y or obj-m (and so on) depending on the value set by that makefile fragment generated by kconfig. The set of object files to build can then be found in "$(obj-y)" (and so on), which is used as a dependency in a makefile rule to link everything together. The same kconfig also creates a C header file with #define rules for these same CONFIG_ symbols, which is passed to the compiler through a "-include" argument. It's not "mostly compiled from a custom KBuild system", IIRC kbuild is the name given to the whole system, including the makefile rules and macros which do the actual compilation.
  - p_l 4y ago
    Decided to go into the source again, and yes, it's mostly implemented in Makefiles, but they are wildly different from the kind of makefile discussed in the article, with considerable amount of machinery in macros and external C programs (whole "features" thing) to the point that individual makefiles in subdird probably wouldn't parse right if not executed through kbuild wrapper

Joker_vD 4y ago

Honestly, I never quite understood the point of generating the makefiles: if you have a tool that knows where the inputs are, where the outputs should be, and tracks the dependencies changing from build to build (something that "make" famously struggles to do), then surely that tool can also run whatever compilers/linkers/etc. are needed? I guess you can commit the generated makefile into your repo so that those who want to build your project from sources don't need your custom build tool installed, only make...

tuyiown 4y ago
Historically, Makefiles generation is merely a cache of platform introspection on systems that were orders of magnitudes slower than what we have now. I suspect that any other approach based on Makefile only would have implied higher memory consumption and/or implied slow rotating disk I/O.
- account42 4y ago
  Actually on my 64-thread Threadripper that cache is more important than ever as the make step parallelizes nicely but the configure step usually is 100% serialized.
rossmohax 4y ago
There are many nuances. How to control parallelism, how to present output, checking freshness, etc. It is much easier to generate Ninja files and let Ninja do all the boring work.
- zelphirkalt 4y ago
  What are the dependencies of Ninja? (I seem to remember it being a Python tool?)
  - flohofwoe 4y ago
    On macOS at least, Ninja is just a 362 kByte native executable without dependencies except dynamically linking with the system-provided libc++ and libSystem DLLs (you're probably thinking of Meson?)

matheusmoreira 4y ago

Like many others, I too rediscovered the power of pure GNU Make years ago. Turns out it has some rather Lisp-like metaprogramming capabilities. It's extremely hard to do even basic processing but with enough effort some cool things can happen. I'm somewhat proud of my implementation of a simple version of the find utility as a pure GNU Make recursive function. This rediscovery essentially derailed one of my side projects as I started writing a pure GNU Make build system and function library like this one: https://gmsl.sourceforge.io/ https://gmsl.sourceforge.io/ https://www.cmcrossroads.com/article/gnu-make-standard-library https://www.cmcrossroads.com/article/gnu-make-standard-libra... I even asked this author on Twitter and Stack Overflow about its implementation. The end result was reinvention of code generation tools because that's what metaprogramming is. I didn't understand what autotools was doing before I tried doing it myself. My objective was to eliminate makefile regeneration whenever project structure changed, I ended up regenerating code 100% of the time and evaluating the resulting code inside the makefile itself. If we must metaprogram, it's better to do it in a proper programming language. At least it didn't have any dependencies other than GNU Make.

rasterdog 4y ago
"Like many others, I too rediscovered the power of pure GNU Make" ... wait a sec, GNU make is not "pure!" I like it but I recognize that it has features that a more portable make would not have.
- matheusmoreira 4y ago
  Perhaps pure was not the right word. By pure GNU Make I meant writing makefiles myself without higher level makefile generation tools.
jgrahamc 4y ago
Hello! That’s me. Happy to answer further questions…
- matheusmoreira 4y ago
  Hello! Huge fan of your work and I've also read your book. Are you still developing GMSL? I wonder if you're open to contributions. My find function has been the single most useful thing to come out of my side project. The first thing I do in my makefiles is get a list of all source files and it's always been annoying because I like organizing my project in trees which requires recursive file system walks. I managed to write a pure GNU Make function that simplifies that to: sources := $(call find,src,file?) The find function recursively walks the src directory and returns a list of all paths satisfying the file? predicate function. I think it would be a nice addition to GMSL. In my project, I doubt it will ever help anyone.

nocman 4y ago

You offer plenty of criticism for using and/or generating makefiles, but I can't find any place in the discussion where you suggest an alternative. So what is your solution for building large projects?