6 ms·

Use `make` to save time

53 points by doomhammer 8y ago

zzo38computer 8y ago
Many other programs use make, and so I use make for compiling those programs. For my own programs I do something different (unless it is a modified version of another program that already uses make). For programs with a single file, I might add something like this at the top: #if 0 gcc -o ~/bin/program program.c exit #endif For programs with multiple files, I will write a shell script to figure out and compile everything (but use stuff like the above to compile each individual file). Although, I should probably make a better way, later.
- palunon 8y ago
  > For programs with multiple files, I will write a shell script to figure out and compile everything (but use stuff like the above to compile each individual file). Although, I should probably make a better way, later. Yeah, you should probably `make` a better way... ;)
- mikekchar 8y ago
  I recommend you spend some time learning how to use make before you spend a great deal of time with your plan. If you do so, once you start trying to rewrite it in a shell script, you will discover pretty quickly why you don't want to. If you don't spend that time, it will take you a lot longer to discover why you shouldn't rewrite it. Either way is likely to have the same result, but one will take less time ;-) To be fair, there are other dependency tools around, but make is a really good starting place to learn what it's about and why you want to use something quite robust. Many people dislike make, but personally I haven't used anything that was substantially better.
finaliteration 8y ago
I’ve been using make for packaging and deploying a variety of AWS CloudFormation stacks and it’s been working pretty well. I use ‘make package’ to generate a final deployment file and run any transforms needed, and then I can run ‘make deploy’ to actually build the stack. It’s been a pretty smooth workflow and I prefer it over the mess of bash scripts we had before (though I know they are different tools with different purposes).
teekert 8y ago
" But show me a better tool for actual development workflow automation and I’ll be glad to switch." Perhaps Snakemake? I've been using it recently in Bioinformatics and I love it. It works perfectly on our Torque/qsub cluster but can also easily switch to a set of dockers in the background.
- doomhammer 8y ago
  Will look at it, thanks!
tralarpa 8y ago
You have noticed a small bug in your data processing script and now you have to redo all your plots for your thesis? latex+gnuplot scripts+makefile, and a few seconds later you have a new pdf.
- doomhammer 8y ago
  That's exactly how I did my thesis ^^
  - majewsky 8y ago
    Same for me, but because I had a lot of TikZ images and pgfplots, I made it so that I could compile each chapter individually to reduce the compile time during editing. Basically, each chapter file would look like: \ifx\ispreambleloaded% \relax% \else% \def\isstandalone{}% \input{preamble}% \fi % ... content ... \ifx\isstandalone\end{document}\fi% and the preamble.tex would look like: \documentclass{scrbook} % ... all the other stuff that goes in the preamble ... \begin{document} \def\ispreambleloaded{}% (Note: Recounting this all from memory, so I may have gotten some stuff wrong.)
drinchev 8y ago
As much as I love unix philosophy and tools that follow it, I would point out that using the build tool that comes with the language is usually the better option. Nobody would use make instead of maven [1] with Java for example. Ruby uses Rakefile instead of Makefiles. You will probably confuse lots of developers that work on your project if you do that. On top of that Makefiles are known to be generated these days ( CMake ). 1 : https://stackoverflow.com/questions/2209827/why-is-no-one-using-make-for-java https://stackoverflow.com/questions/2209827/why-is-no-one-us...
- tomohawk 8y ago
  Who would use maven if they could use buildr? It's much faster and the build files are much easier to understand.
  - kasperni 8y ago
    https://www.jetbrains.com/research/devecosystem-2018/java/ https://www.jetbrains.com/research/devecosystem-2018/java/ Apparently, most people......
- doomhammer 8y ago
  I agree, notice that I use `make` together with `npm` as an example. Or with `setup.py`. Language build tools are great, but `make` comes in when you need to tie them together.
- chungy 8y ago
  Maven does not come with Java, and OpenJDK itself is built with Make rather than any other tool. The idea that "nobody" uses make for Java when Java itself does, is a little bit a stretch ;)
- charleslmunger 8y ago
  Maven is possibly one of the worst build systems I've used. It's an imperative build system with a declarative syntax; not parallel by default; always requires a clean build in practice, unless your code is all stuffed into one big blob. For a long time, it happily downloaded executable code from the internet without even basic cryptographic verification. Even the ergonomics of the tool are poor - it spams log output with all sorts of information other than what you actually need (like the output of a failed compilation). Getting information from a failed test is painful. If you change one module that another depends on, the latter will not be rebuilt - letting you find issues via NoSuchMethodError at runtime. Annotation processors require special care to enable, and don't always run when they should. I would rather use make and javac than Maven. Gradle, Bazel (and its diaspora of non-google imitations) would be much better choices.
regularfry 8y ago
I think there's a bug in this page, I'm not seeing code snippets that are referred to in the text. Shame, because I want to point coworkers at it.
- doomhammer 8y ago
  You are right! I notified the editors to learn why the code got lost during editing.
bloak 8y ago
"Make" is great for small projects. But as the project grows people start to fill the Makefile with hacks: all those empty files with "stamp" in the name and scripts called "move-if-change" and other tricks that don't quite follow the rules. And recursive "make" invocations (http://aegis.sourceforge.net/auug97.pdf http://aegis.sourceforge.net/auug97.pdf). Eventually the whole thing becomes so incomprehensible and unreliable that the general advice becomes: do a fresh a checkout before running "make", or build out-of-tree and do "rm -r *" first. You almost certainly have to do that if the stuff you're trying to debug is the build infrastructure itself, which, sadly, is something developers spend an awful lot of time doing. Don't get me started on Makefiles that invoke Docker. Some practical advice: don't write a rule like "./script < input > output"; instead write "./script < input > output.new && mv output.new output". That way if the script fails it doesn't leave things in a broken state with "output" appearing to be up-to-date. Perhaps a replacement for "make" would incorporate this mechanism? A replacement for "make" would also have a proper way of handling the situation in which a file is regenerated and found not to have changed, to replace all those half-working "stamps" and "move-if-change" hacks. Edit: See also: https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html https://www.gnu.org/software/automake/manual/html_node/Multi... Presumably a decent build system would make this fairly normal situation somewhat easier to handle.
- JdeBP 8y ago
  They already do. * http://jdebp.eu./FGA/introduction-to-redo.html http://jdebp.eu./FGA/introduction-to-redo.html (https://news.ycombinator.com/item?id=18398957 https://news.ycombinator.com/item?id=18398957) * https://news.ycombinator.com/item?id=18480433 https://news.ycombinator.com/item?id=18480433 * https://news.ycombinator.com/item?id=18473744 https://news.ycombinator.com/item?id=18473744
- de_watcher 8y ago
  Oh, Docker is especially garbage to automate with usual tools. You have to write an entire application via the API to do stuff somewhat reliably.
- ynik 8y ago
  > That way if the script fails it doesn't leave things in a broken state with "output" appearing to be up-to-date. Perhaps a replacement for "make" would incorporate this mechanism? GNU make has ".DELETE_ON_ERROR" for this. But yes, make is quite horrible if you want reliable incremental builds. For example, $OLD_VERSION of your software generates a file $output/plugins/stuff.so, and $NEW_VERSION no longer does so, the file will stick around and the new version of the software will load an outdated plugin. This type of stuff causes weird issues that disappear on the next fresh checkout (especially during development, where the plugin ABI may be unstable). If a rule disappears, so should its output file(s) on the next build. Why are there so many build systems that don't even attempt to handle this stuff? Surely we aren't the only project that uses glob patterns like plugins/*.so? You can't fix these fundamental design problems by using meta-build-tools that end up generating makefiles...
tomohawk 8y ago
I used make for several years, but switched to rake and never looked back. Martin Fowler put together a nice article about rake: https://martinfowler.com/articles/rake.html https://martinfowler.com/articles/rake.html If you deal a lot with artifacts you need to download from random places (ala maven, but with a lot more flexibility), or produce artifacts to various repos, buildr is very nice.
- doomhammer 8y ago
  Yes, but this requires Ruby which may pose some problems. I am well aware of rake and like it very much. But I wouldn't consider it a proper replacement for make.
pmoriarty 8y ago
As a user, what I really miss in a lot of make replacements is an obvious way to install a custom location. Something like ./cofigure --prefix=/usr/local/foo Another thing I miss is the ability to easily pass in custom compilation options, like: CFLAGS="-O0 -ggdb3" LDFLAGS="-L/usr/local/foo/include" ./configure I know these are features of autotools I'm talking about, not of make per se. Still, a lot of tools that are supposed to be way better than make and autotools just seem to assume every user's going to install in the same location and offer no obvious way to give different compilation options and as a user it's really frustrating because I need both all the time.
- yaris 8y ago
  My limited experience with SCons suggests that it is not worse w.r.t this functionality, assuming both configure/make source for autotools and SConstrict/SConscript are written correctly.
riskneutral 8y ago
Why would anyone advise using GNU make instead of cmake? It is well known that make itself is insufficient for complex builds. This led to development of meta tools like GNU autotools and cmake. I would argue that cmake is better and saves time over using autotools. That said, I am really impressed with Google’s meta tool called GN which generates scripts for a build system called Ninja. This tool chain is based on performance, and the scripting is much easy/less ugly than cmake.
- haolez 8y ago
  You don’t understand what make is for. It doesn’t compete with CMake. It’s actually one of CMake’s backends.
  - chillacy 8y ago
    I think the parent poster calling CMake a meta tool might imply they knew that.
    - makapuf 8y ago
      I guess so. Generally you write either one or the other. If you write cmakelists, make is just a subtool, you never quite see makefiles. So for the computer sure both run makefikes. From the developer point of view it's not.
haolez 8y ago
Make is an amazing tool! It gives you a lot of power with just a few new concepts to learn. However, I don’t see much value in using make as intended in the article. The author uses it as some sort of “shared bash history” for developers, where it is used to store the most relevant commands for a project. Make is good for keeping track of what changed in your environment and, when it does change, to do the least amount of work necessary to build your asset. For the problem that the author is trying to solve, I would recommend something like Ansible, which feels simple and its playbooks often serve as documentation.
effnorwood 8y ago
Try ‘quit’
stephenr 8y ago
My only gripe about make (I use gnu make almost daily) is so far I haven’t discovered a good way to test posix make functionality. The .POSIX special target doesn’t really make gnu make follow posix strictly.

53 points by doomhammer 8y ago

Many other programs use make, and so I use make for compiling those programs. For my own programs I do something different (unless it is a modified version of another program that already uses make). For programs with a single file, I might add something like this at the top: #if 0 gcc -o ~/bin/program program.c exit #endif For programs with multiple files, I will write a shell script to figure out and compile everything (but use stuff like the above to compile each individual file). Although, I should probably make a better way, later.

palunon 8y ago
> For programs with multiple files, I will write a shell script to figure out and compile everything (but use stuff like the above to compile each individual file). Although, I should probably make a better way, later. Yeah, you should probably `make` a better way... ;)
mikekchar 8y ago
I recommend you spend some time learning how to use make before you spend a great deal of time with your plan. If you do so, once you start trying to rewrite it in a shell script, you will discover pretty quickly why you don't want to. If you don't spend that time, it will take you a lot longer to discover why you shouldn't rewrite it. Either way is likely to have the same result, but one will take less time ;-) To be fair, there are other dependency tools around, but make is a really good starting place to learn what it's about and why you want to use something quite robust. Many people dislike make, but personally I haven't used anything that was substantially better.

finaliteration 8y ago

I’ve been using make for packaging and deploying a variety of AWS CloudFormation stacks and it’s been working pretty well. I use ‘make package’ to generate a final deployment file and run any transforms needed, and then I can run ‘make deploy’ to actually build the stack. It’s been a pretty smooth workflow and I prefer it over the mess of bash scripts we had before (though I know they are different tools with different purposes).

teekert 8y ago

" But show me a better tool for actual development workflow automation and I’ll be glad to switch." Perhaps Snakemake? I've been using it recently in Bioinformatics and I love it. It works perfectly on our Torque/qsub cluster but can also easily switch to a set of dockers in the background.

doomhammer 8y ago
Will look at it, thanks!

tralarpa 8y ago

You have noticed a small bug in your data processing script and now you have to redo all your plots for your thesis? latex+gnuplot scripts+makefile, and a few seconds later you have a new pdf.

doomhammer 8y ago
That's exactly how I did my thesis ^^
- majewsky 8y ago
  Same for me, but because I had a lot of TikZ images and pgfplots, I made it so that I could compile each chapter individually to reduce the compile time during editing. Basically, each chapter file would look like: \ifx\ispreambleloaded% \relax% \else% \def\isstandalone{}% \input{preamble}% \fi % ... content ... \ifx\isstandalone\end{document}\fi% and the preamble.tex would look like: \documentclass{scrbook} % ... all the other stuff that goes in the preamble ... \begin{document} \def\ispreambleloaded{}% (Note: Recounting this all from memory, so I may have gotten some stuff wrong.)

drinchev 8y ago

As much as I love unix philosophy and tools that follow it, I would point out that using the build tool that comes with the language is usually the better option. Nobody would use make instead of maven [1] with Java for example. Ruby uses Rakefile instead of Makefiles. You will probably confuse lots of developers that work on your project if you do that. On top of that Makefiles are known to be generated these days ( CMake ). 1 : https://stackoverflow.com/questions/2209827/why-is-no-one-using-make-for-java https://stackoverflow.com/questions/2209827/why-is-no-one-us...

tomohawk 8y ago
Who would use maven if they could use buildr? It's much faster and the build files are much easier to understand.
- kasperni 8y ago
  https://www.jetbrains.com/research/devecosystem-2018/java/ https://www.jetbrains.com/research/devecosystem-2018/java/ Apparently, most people......
doomhammer 8y ago
I agree, notice that I use `make` together with `npm` as an example. Or with `setup.py`. Language build tools are great, but `make` comes in when you need to tie them together.
chungy 8y ago
Maven does not come with Java, and OpenJDK itself is built with Make rather than any other tool. The idea that "nobody" uses make for Java when Java itself does, is a little bit a stretch ;)
charleslmunger 8y ago
Maven is possibly one of the worst build systems I've used. It's an imperative build system with a declarative syntax; not parallel by default; always requires a clean build in practice, unless your code is all stuffed into one big blob. For a long time, it happily downloaded executable code from the internet without even basic cryptographic verification. Even the ergonomics of the tool are poor - it spams log output with all sorts of information other than what you actually need (like the output of a failed compilation). Getting information from a failed test is painful. If you change one module that another depends on, the latter will not be rebuilt - letting you find issues via NoSuchMethodError at runtime. Annotation processors require special care to enable, and don't always run when they should. I would rather use make and javac than Maven. Gradle, Bazel (and its diaspora of non-google imitations) would be much better choices.

regularfry 8y ago

I think there's a bug in this page, I'm not seeing code snippets that are referred to in the text. Shame, because I want to point coworkers at it.

doomhammer 8y ago
You are right! I notified the editors to learn why the code got lost during editing.

bloak 8y ago

"Make" is great for small projects. But as the project grows people start to fill the Makefile with hacks: all those empty files with "stamp" in the name and scripts called "move-if-change" and other tricks that don't quite follow the rules. And recursive "make" invocations (http://aegis.sourceforge.net/auug97.pdf http://aegis.sourceforge.net/auug97.pdf). Eventually the whole thing becomes so incomprehensible and unreliable that the general advice becomes: do a fresh a checkout before running "make", or build out-of-tree and do "rm -r *" first. You almost certainly have to do that if the stuff you're trying to debug is the build infrastructure itself, which, sadly, is something developers spend an awful lot of time doing. Don't get me started on Makefiles that invoke Docker. Some practical advice: don't write a rule like "./script < input > output"; instead write "./script < input > output.new && mv output.new output". That way if the script fails it doesn't leave things in a broken state with "output" appearing to be up-to-date. Perhaps a replacement for "make" would incorporate this mechanism? A replacement for "make" would also have a proper way of handling the situation in which a file is regenerated and found not to have changed, to replace all those half-working "stamps" and "move-if-change" hacks. Edit: See also: https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html https://www.gnu.org/software/automake/manual/html_node/Multi... Presumably a decent build system would make this fairly normal situation somewhat easier to handle.

JdeBP 8y ago
They already do. * http://jdebp.eu./FGA/introduction-to-redo.html http://jdebp.eu./FGA/introduction-to-redo.html (https://news.ycombinator.com/item?id=18398957 https://news.ycombinator.com/item?id=18398957) * https://news.ycombinator.com/item?id=18480433 https://news.ycombinator.com/item?id=18480433 * https://news.ycombinator.com/item?id=18473744 https://news.ycombinator.com/item?id=18473744
de_watcher 8y ago
Oh, Docker is especially garbage to automate with usual tools. You have to write an entire application via the API to do stuff somewhat reliably.
ynik 8y ago
> That way if the script fails it doesn't leave things in a broken state with "output" appearing to be up-to-date. Perhaps a replacement for "make" would incorporate this mechanism? GNU make has ".DELETE_ON_ERROR" for this. But yes, make is quite horrible if you want reliable incremental builds. For example, $OLD_VERSION of your software generates a file $output/plugins/stuff.so, and $NEW_VERSION no longer does so, the file will stick around and the new version of the software will load an outdated plugin. This type of stuff causes weird issues that disappear on the next fresh checkout (especially during development, where the plugin ABI may be unstable). If a rule disappears, so should its output file(s) on the next build. Why are there so many build systems that don't even attempt to handle this stuff? Surely we aren't the only project that uses glob patterns like plugins/*.so? You can't fix these fundamental design problems by using meta-build-tools that end up generating makefiles...

tomohawk 8y ago

I used make for several years, but switched to rake and never looked back. Martin Fowler put together a nice article about rake: https://martinfowler.com/articles/rake.html https://martinfowler.com/articles/rake.html If you deal a lot with artifacts you need to download from random places (ala maven, but with a lot more flexibility), or produce artifacts to various repos, buildr is very nice.

doomhammer 8y ago
Yes, but this requires Ruby which may pose some problems. I am well aware of rake and like it very much. But I wouldn't consider it a proper replacement for make.

pmoriarty 8y ago

As a user, what I really miss in a lot of make replacements is an obvious way to install a custom location. Something like ./cofigure --prefix=/usr/local/foo Another thing I miss is the ability to easily pass in custom compilation options, like: CFLAGS="-O0 -ggdb3" LDFLAGS="-L/usr/local/foo/include" ./configure I know these are features of autotools I'm talking about, not of make per se. Still, a lot of tools that are supposed to be way better than make and autotools just seem to assume every user's going to install in the same location and offer no obvious way to give different compilation options and as a user it's really frustrating because I need both all the time.

yaris 8y ago
My limited experience with SCons suggests that it is not worse w.r.t this functionality, assuming both configure/make source for autotools and SConstrict/SConscript are written correctly.

riskneutral 8y ago

Why would anyone advise using GNU make instead of cmake? It is well known that make itself is insufficient for complex builds. This led to development of meta tools like GNU autotools and cmake. I would argue that cmake is better and saves time over using autotools. That said, I am really impressed with Google’s meta tool called GN which generates scripts for a build system called Ninja. This tool chain is based on performance, and the scripting is much easy/less ugly than cmake.

haolez 8y ago
You don’t understand what make is for. It doesn’t compete with CMake. It’s actually one of CMake’s backends.
- chillacy 8y ago
  I think the parent poster calling CMake a meta tool might imply they knew that.
  - makapuf 8y ago
    I guess so. Generally you write either one or the other. If you write cmakelists, make is just a subtool, you never quite see makefiles. So for the computer sure both run makefikes. From the developer point of view it's not.

haolez 8y ago

Make is an amazing tool! It gives you a lot of power with just a few new concepts to learn. However, I don’t see much value in using make as intended in the article. The author uses it as some sort of “shared bash history” for developers, where it is used to store the most relevant commands for a project. Make is good for keeping track of what changed in your environment and, when it does change, to do the least amount of work necessary to build your asset. For the problem that the author is trying to solve, I would recommend something like Ansible, which feels simple and its playbooks often serve as documentation.

effnorwood 8y ago

Try ‘quit’

stephenr 8y ago

My only gripe about make (I use gnu make almost daily) is so far I haven’t discovered a good way to test posix make functionality. The .POSIX special target doesn’t really make gnu make follow posix strictly.