3 ms·

> it relies on timestamps instead of content hashes If it relied upon content hashes, then make would have to hash the files each time it ran (taking time---ch

by spc476 10y ago

> it relies on timestamps instead of content hashes If it relied upon content hashes, then make would have to hash the files each time it ran (taking time---checking the timestamp is faster). For a large project, this might be too excessive. > it doesn't rebuild things when e.g. compiler flags change Just make the Makefile a dependency on each target. That problem solved. > it needs to parse the Makefile each time Um ... of course. The only way around this is for make to compile into some easier format to read or some database type thing (perhaps a .makefile?) and it would need to track changes to Makefile to update the database type thing (well, make does dependency tracking so that shouldn't be too hard). Just a question---do you have a project where reading the Makefile takes a non-negligible time? > it doesn't have an integration with inotify inotify is a) Linux only (make, and specifically GNU Make, runs on nearly everything) and b) it's an API, not a service that can be queried. Doing this implies that make will have to always be running checking all files for a project. Which project? All projects that have a makefile on my system? After a certain number of files being tracked, I'm certain that checking "inotify" (a daemon perhaps?) is the same as checking "stat()" for file metadata.

paulddraper 10y ago
These are solvable approaches, though it's easy to see why make did what it did. > hash the files each time it ran Bazel & co relies on file hashes, but only checks then when the timestamp changes. Downside: now you have to keep hashes as metadata somewhere. > it needs to parse the Makefile each time Many build tools run a long running process either in the foreground (SBT, new Maven) or in the background (Gradle). Downside: Now you have to run a long running process... > it doesn't have an integration with inotify It could. Downside: Not universal, uses memory, more complex.
pg314 10y ago
> If it relied upon content hashes, then make would have to hash the files each time it ran (taking time---checking the timestamp is faster). For a large project, this might be too excessive. You can combine timestamps with content hashes. Or if you have inotify integration, you only need to process a file when it changes. > Just make the Makefile a dependency on each target. That problem solved. You can override variables when invoking make without changing the Makefile. Problem not solved. Of course there are some hacks to add dependencies on variable values. The majority of Makefiles in the wild don't do this, requiring you to do 'make clean; make' if you change flags. Defaults matter. > Just a question---do you have a project where reading the Makefile takes a non-negligible time? At the moment, no. But just to give you an idea of how it could become a problem: a simple C++ file including <iostream> leads to a 25KB .d file with 200 dependencies (generated by clang++ -MD). For a project with 1000 files, I made a 25MB file with one target and 20000 dependencies on the same file. Make took one second to process that. That is about the simplest Makefile to read. Throw in some includes, multiple Makefiles, variable expansions, and external shell invocations and that time will go up. > inotify is a) Linux only (make, and specifically GNU Make, runs on nearly everything) I should have said inotify or similar system (bsd has kqueue, I imagine Windows has something similar). > b) it's an API, not a service that can be queried. Doing this implies that make will have to always be running checking all files for a project. Which project? The one I'm working on. I'll start a daemon when I'm working on a project. As long as we're dreaming, it would be nice to have a filesystem that integrates Merkle trees and exposes an API you could query to see if anything has changed in a subdirectory. Apple kludged something together with their file systems events API [1] that provides similar functionality. It provides an API to query what has changed in a directory subtree since a previous invocation. No need for a persistent process. They use it for their Time Machine backup software. > After a certain number of files being tracked, I'm certain that checking "inotify" (a daemon perhaps?) is the same as checking "stat()" for file metadata. No. With inotify the work is proportional to the number of changed files. With stat() you need to check every file, so the work is proportional to the total number of files in your project. As I said elsewhere in this thread, most of the design decisions of Make make sense given its history. That doesn't mean they are still optimal today. [1] https://developer.apple.com/library/content/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html https://developer.apple.com/library/content/documentation/Da...
LukeShu 10y ago
> Just a question---do you have a project where reading the Makefile takes a non-negligible time? I have a project where reading the Makefile (and makefiles `include`d by it) takes about 12 seconds. The little profiling that I've done leads me to believe that most of it is spent in the garbage collector. Regardless of whether that is true, for Makefiles that take a long time to "parse", the time isn't spent in parsing per se, it's spent managing the datastructures that parsing creates (or spent evaluating expensive things like $(shell) that because of lazy evaluation end up getting evaluated many times; but that's a smell of a poorly written Makefile). > Um ... of course. The only way around this is for make to compile into some easier format to read or some database type thing (perhaps a .makefile?) That is what ninja was designed for.

by spc476 10y ago

paulddraper 10y ago

These are solvable approaches, though it's easy to see why make did what it did. > hash the files each time it ran Bazel & co relies on file hashes, but only checks then when the timestamp changes. Downside: now you have to keep hashes as metadata somewhere. > it needs to parse the Makefile each time Many build tools run a long running process either in the foreground (SBT, new Maven) or in the background (Gradle). Downside: Now you have to run a long running process... > it doesn't have an integration with inotify It could. Downside: Not universal, uses memory, more complex.

pg314 10y ago

> If it relied upon content hashes, then make would have to hash the files each time it ran (taking time---checking the timestamp is faster). For a large project, this might be too excessive. You can combine timestamps with content hashes. Or if you have inotify integration, you only need to process a file when it changes. > Just make the Makefile a dependency on each target. That problem solved. You can override variables when invoking make without changing the Makefile. Problem not solved. Of course there are some hacks to add dependencies on variable values. The majority of Makefiles in the wild don't do this, requiring you to do 'make clean; make' if you change flags. Defaults matter. > Just a question---do you have a project where reading the Makefile takes a non-negligible time? At the moment, no. But just to give you an idea of how it could become a problem: a simple C++ file including <iostream> leads to a 25KB .d file with 200 dependencies (generated by clang++ -MD). For a project with 1000 files, I made a 25MB file with one target and 20000 dependencies on the same file. Make took one second to process that. That is about the simplest Makefile to read. Throw in some includes, multiple Makefiles, variable expansions, and external shell invocations and that time will go up. > inotify is a) Linux only (make, and specifically GNU Make, runs on nearly everything) I should have said inotify or similar system (bsd has kqueue, I imagine Windows has something similar). > b) it's an API, not a service that can be queried. Doing this implies that make will have to always be running checking all files for a project. Which project? The one I'm working on. I'll start a daemon when I'm working on a project. As long as we're dreaming, it would be nice to have a filesystem that integrates Merkle trees and exposes an API you could query to see if anything has changed in a subdirectory. Apple kludged something together with their file systems events API [1] that provides similar functionality. It provides an API to query what has changed in a directory subtree since a previous invocation. No need for a persistent process. They use it for their Time Machine backup software. > After a certain number of files being tracked, I'm certain that checking "inotify" (a daemon perhaps?) is the same as checking "stat()" for file metadata. No. With inotify the work is proportional to the number of changed files. With stat() you need to check every file, so the work is proportional to the total number of files in your project. As I said elsewhere in this thread, most of the design decisions of Make make sense given its history. That doesn't mean they are still optimal today. [1] https://developer.apple.com/library/content/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html https://developer.apple.com/library/content/documentation/Da...

LukeShu 10y ago

> Just a question---do you have a project where reading the Makefile takes a non-negligible time? I have a project where reading the Makefile (and makefiles `include`d by it) takes about 12 seconds. The little profiling that I've done leads me to believe that most of it is spent in the garbage collector. Regardless of whether that is true, for Makefiles that take a long time to "parse", the time isn't spent in parsing per se, it's spent managing the datastructures that parsing creates (or spent evaluating expensive things like $(shell) that because of lazy evaluation end up getting evaluated many times; but that's a smell of a poorly written Makefile). > Um ... of course. The only way around this is for make to compile into some easier format to read or some database type thing (perhaps a .makefile?) That is what ninja was designed for.