11 ms·
Notes for new Make users
- lolikoisuru 9y agoIf you’re feeling mischievous, Make has divers ‘secret’ vars & targets for you. For instance, you don’t have to ‘struggle’ w/ the ancient bash, for it’s possible to program recipes in any language that supports evaluations straight from the command line. In addition you can have different shell for each rule with the rule specific macros if you do desire: .ONESHELL: .SILENT: .PHONY: rule1 rule1: SHELL := /usr/bin/python3 rule1: foo=bar print("python: {}".format(foo))
- lolikoisuru 9y ago>When in doubt, don’t google until you read the official manual. It’s a document of an astonishing quality. Go print the pdf. This applies to most GNU software.
- lolikoisuru 9y agoOne thing I sometimes wish was possible in make is using the automatic variables in the dependency part of the rules. The one that I most often have desired to use is $(@D) so I could depend on a file with specific name in the same directory when the directory name isn't known beforehand. I got around this with some touch trickery but it would be just easier to read if those automatic variables were available at that point.
- setheron 9y agoGreat guide. The variable portion was the first time it was explained succinctly and simply to me.
- henry_flower 9y agothanks!
- ISL 9y agoThe point about the manual is a good one. It is well done. When I made extensive use of Make for my graduate thesis, I bought the book and read the majority of the text. The cover illustration brings me joy every time I see or think about it.
- deleted 9y ago[deleted]
- nwellnhof 9y agoI'd suggest to read the POSIX man page first: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ma... POSIX make is rather limited but if you start to rely on GNU make features, you'll end up bound to a single implementation.
- EpicEng 9y ago>you'll end up bound to a single implementation. Which will be just fine for something like 99% of us.
- wahern 9y agoPOSIX make has the virtue of simplicity. GNU make : POSIX make :: C++ : C. It's even worse than that because the way GNU make extends the syntax is completely at odds with core make syntax. It's why to write a looping construct in GNU make you need the contortion of using eval, whereas a loop in the BSD family of makes is much more simple and concise. I'm not advocating to use a BSD make, nor saying not to use GNU make. But appreciating GNU make's points of departure from POSIX make helps keep your makefiles simple and to the point. Which is important because another virtue of make generally (GNU make included) is that it'll likely still be around long after all these alternatives have been long forgotten. But time is never kind to complexity.
- dblotsky 9y agoThis is so excellent. I was smiling as I read it. Thank you. My personal favourite "watcher" implementation is this: while true; do make; sleep 0.5; done
- FrenchyJiby 9y agoI don't understand: why not the classic `watch make` (watch defaults to 2 seconds)?
- gengen 9y agoWhat I've always loved about Make and related tools is how directly they expose their fundamental concept in building artefacts: the DAG. I don't do that much JS, but I've not found this kind of clarity in any of the JS frontend build tools I've used. Honest question for the webpack power-users here. What are some important webpack features that you lose by using a Makefile like this? Because it seems like pretty much every part of a regular webpack buildchain has a command-line equivalent that you could potentially use in a Makefile.
- sillysaurus3 9y agoYou lose the simplicity of copy-pasting someone else's webpack config and then morphing it into something that works for your project. There are tens of thousands of examples to choose from. For a makefile-based webpack, there are zero. Unfortunately concerns like this matter.
- gkya 9y agoWhat does building a JS app take? Linting maybe, then concatenating files, minifying them. Then there should be some maintaining tasks, like updating dependencies, testing, deploying. Am I missing anything (serious question, I'm rather unfamiliar)?
- ino 9y agoMinifiers can remove dead and unaccessible code. They can also rename function and variable names with shorter ones, and compile code to be shorter (marginal gains since it's all going to be served compressed).
- markmark 9y agoI don't know enough about make to know if this is correct, but the post last week on using make for js started of by saying "If you are writing a frontend app and you need code bundling you should absolutely use Webpack" which, aren't the majority of people using webpack writing frontend apps and need bundling? The other issue is that essentially no one uses make in frontend development. So you go and convert to make from something that was working fine and is the standard in that part of the industry and what's the pay off? Now no one in your team or who you hire knows how to change the build chain? Great.
- andrewmcwatters 9y agoThe first point is "Don’t try to be clever," and using Make in the JavaScript ecosystem defies exactly that to begin with.
- pvtmert 9y agoit is called 'build system' so is gradle, maven, xcode, it does not matter what target is or what source is (as language, file etc) they have tasks/recipes and dependencies/prerequistes imho JS's stack of build system still not good at all. changing to new system every 2-3 years. (note that i am not a js-dev, just outside point of view) which is kind of a not-stable-ecosystem feeling. I see small agencies spending days maybe weeks to port their 'themes' to new stacks (long ago jquery, then gulp, now webpack) where as haven't encountered single "porting to new makefile format" case, because, of course solid history/background of make.
- andrewmcwatters 9y agoyeah what's an even better idea is not to introduce a new build system, but to introduce an old one primarily used by an entirely different language ecosystem brilliant
- vorg 9y ago> gradle, maven, xcode, it does not matter what target is or what source is (as language But it does matter what the language is for the build script for each build system, e.g. Apache Groovy for Gradle, XML for Maven.
- bringtheaction 9y agoNo it doesn’t. Makefiles define inputs, outputs and the commands that transform input to output. Make then compares the last modified timestamp of inputs with outputs on each run and reruns the commands for the targets that are out of date. Make is in no way limited to C or C++ projects and there is absolutely nothing wrong in using Make for a JavaScript project. The only thing that sucks a bit is maintaining the list of inputs and outputs if you have a lot of files. Well, that and the fact that Makefiles are not generally portable between GNU and BSD Make so if you use both Linux and a BSD you need to install either gmake on BSD or bmake on Linux and then remember to invoke that rather than just make for your project. Of course you could have a bash function that will look and pick GNU make if there is a GNUmakefile in your directory and BSD make if there is a BSDmakefile, but maintaining such things is a hassle as well. It always comes down to picking the right tool and sometimes Make will be it and sometimes it won’t but whether the output is JavaScript or a PDF (e.g. from LaTeX documents) or JPEGs or whatever has very little to do with it.
- pspeter3 9y agoIs it possible to have make use file hashes instead of timestamps?
- gkya 9y agoThere is DJB's redo[1], which has some implementations. Also, one could probably modify make to use hashes. But personally I think that mtime is a nice compromise, as it's often a correct heuristic, and hashing files is not cheap, especially for big projects that'd have a performance penalty. [1] https://cr.yp.to/redo.html https://cr.yp.to/redo.html
- henry_flower 9y agoin theory, yes: https://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes https://www.cmcrossroads.com/article/rebuilding-when-files-c... but I've never really tried to use such a hack
- vortico 9y agoPretty spot on. I use a pretty similar dependency system for all of my C and C++ projects. FLAGS += -MMD CFLAGS += $(FLAGS) CXXFLAGS += $(FLAGS) OBJECTS += $(patsubst %, build/%.o, $(SOURCES)) DEPS = $(patsubst %, build/%.d, $(SOURCES)) $(TARGET): $(OBJECTS) $(CXX) -o $@ $^ $(LDFLAGS) -include $(DEPS) build/%.c.o: %.c @mkdir -p $(@D) $(CC) $(CFLAGS) -c -o $@ $< build/%.cpp.o: %.cpp @mkdir -p $(@D) $(CXX) $(CXXFLAGS) -c -o $@ $<
- babuskov 9y agoThis looks very similar to the Makefile I'm using except that beside -MMD I also have -MP flag. I created this Makefile a couple years back and I'm reusing it for every new project, but I cannot recall really why I needed to add -MP.
- vortico 9y agohttps://gcc.gnu.org/onlinedocs/gcc-4.8.0/gcc/Preprocessor-Options.html https://gcc.gnu.org/onlinedocs/gcc-4.8.0/gcc/Preprocessor-Op... Looks useful, I'll add it to my configs.
- babuskov 9y agoYes, that's it. Removing a .h file from your project would trigger errors without that switch. I looked at clang manual when I searched for it, but unlike GCC it does not explain it.
- cozzyd 9y agoWow, that -MMD flag is very handy. Can't believe I didn't know about it until now.
- v_lisivka 9y agoHow to use it?
- vortico 9y ago
- elsherbini 9y agoI use snakemake[0] as a replacement for make. The debugging tools in snakemake are really good - it can output the DAG of your workflow as a dotfile, and you can add breakpoints and use pdb to debug your workflow. It also integrates really nicely with conda, so you can have different environments for different rules. The killer feature of snakemake for me is that it can run a workflow on a single core or as many as you give it, or in the case of a cluster it can actually submit jobs to a queue and wait for them to finish before submitting more. The tradeoff is that you need to install it (conda install -c bioconda snakemake), and that it can be very slow if you have thousands of target files. But for most workflows it's the right tool for me. [0] http://snakemake.readthedocs.io/en/stable/ http://snakemake.readthedocs.io/en/stable/
- blcArmadillo 9y ago> The killer feature of snakemake for me is that it can run a workflow on a single core or as many as you give it, or in the case of a cluster it can actually submit jobs to a queue and wait for them to finish before submitting more. Other than your cluster example isn't this the same as passing -j to make to run jobs in parallel?
- rflrob 9y agoYeah, -j just about does it, though with Snakemake you can also specify other resources (memory, I/o, etc) for each task, and it will keep all of them under quota. So if you tell snakemake that task A takes six cores, but B-K take only one each, it will solve the knapsack problem to come up with some kind of optimal order for them, and on your 8-core machine you could have A, B, and C running, or B-K. For a while, I muddled through with Make and a custom cluster job-submitter script, but the real Snakemake killer feature for me was multiple, independent wildcards. So I can, in one rule, have dir/{databasename}/{query}.tsv and rapidly hit multiple databases at once. If you are very clever (and rule one of Make is don’t be clever), you can have a GNUMake wildcard get parsed into multiple wildcards with the wimpy pattern substitution rules, but this is not a fun way to relax.
- 9y ago
- jstimpfle 9y agoI've been working on a small custom build system (of the "re-editable" kind) for my own software project, and have been working on a little writeup. It's not finished, and I think there was something in the implementation that I was still unhappy about. But I'll take this opportunity to jump on the bandwagon. I think for larger project it is worth it to just make your own system, so here are my notes: http://jstimpfle.de/blah/buildsystem.html http://jstimpfle.de/blah/buildsystem.html
- jumpinalake 9y agoI'm having trouble reading past "Go print the pdf". It's 220 pages.
- LinXitoW 9y agoIn all these discussions about make, everyone argues about using it for everything. Personally, I don't know another tool that makes it straight forward to glue together all kinds of different build tools (for example webpack for frontend and maven/sbt for the backend). Even if you don't directly use make, it makes for great, verifiable documentation.
- mahoro 9y agoGreat article. Also, author prepared a surprice for all coming to his blog from russian IP addresses. That was the first time I came across such thing.
- confounded 9y agoCan you explain it for us?
- Igor_kh 9y agoThe author has put an offensive banner (https://imgur.com/a/84cuN https://imgur.com/a/84cuN) in front of his blog page for Russian IP addresses. I just wonder why there is no such thing in the article above
- henry_flower 9y agothanks!
- vorotato 9y agoThe only reason I don't use it is because I use windows. While you can theoretically use it on windows, nobody does.
- dahart 9y ago> While you can theoretically use it on windows, nobody does. What do you use instead? I think a lot of people use make on Windows. I've had several jobs with a Windows dev environment, and they've all used make. My current dev env is cmake on Windows, so I don't touch the makefiles, but it's still using make on Windows. There aren't that many options for cross-platform projects. A lot of people use cygwin or the Windows Subsystem for Linux with make and/or cmake native. I also go directly to make on either cygwin or unix for one-off builds all the time, in non-C/C++ projects. make is super handy for resizing images or running a batch of tests on a program generating graphs. Anything where you need to generate files that only update when another file is touched is a good job for make. make -j is extra handy on windows for some situations since it's hard to get gnu parallel to run in cygwin and the parallel perf wasn't very good last time I checked.
- skocznymroczny 9y agoMost Windows projects that I've seen in the open use CMake to create VS solutions which usually just work.
- orthecreedence 9y agoI do. I live in MSYS2 on Windows. Can't stand VS or powershell, and I've invested so heavily in the GNU toolchain over the years that having the option on Windows is a no-brainer for me. I use Make for everything from frontend builds (don't like grunt/gulp) to rust apps. The better I get at Make, the more I love it.
- abhishekjha 9y agoWait. Did I read it wrong or didn't an article a few days back say that we can use make with file names consisting of spaces, its just that we shouldn't? Although that article talked about using a non-breaking space as a hack.
- spc476 9y agoMake doesn't handle filenames with spaces (ASCII 32) at all! I wrote the article (http://boston.conman.org/2018/02/28.2 http://boston.conman.org/2018/02/28.2) about using the non-breaking space (NBSP, Unicode 160) in filenames, and make (as well as nearly everything else I tried) accepted it just fine. But popular consensus (from here and Lobsters) seemed to be "avoid any type of space at all cost and do not speak of it again!". Pity.
- abhishekjha 9y agoOh yes. I got so pumped up by your article(and partially by losters' comments of never using it again) that I went on my own to replicate the scenario. Only then I was able to understand what you meant by : > It also requires a filesystem that can support Unicode (or in my case, UTF-8) and a command line that also supports Unicode (or again in my case, UTF-8). > And it was also not that easy to create the filename and Makefile with the non-breaking space.. Looks like setting up the compose key on linux did the trick. I posted my own recipe for naming files using non-breaking spaces[0] after someone asked for it. A beginner here so the steps may look a little too primitive but it gets the work done. Thanks for introducing me to a hack. Now I can have a folder with two files whose name look exactly the same(ASCII 32, NBSP). [0]https://news.ycombinator.com/item?id=16494394 https://news.ycombinator.com/item?id=16494394
- dasyatidprime 9y agoI think the main point of handling file names with character code 0x20 in them is for situations where you don't get to pick the file names. I have a project in which I'm juggling a large amount of files imported from external sources, which need to be variably processed and used as-is and in either case must have their names preserved. I wanted to use Make to handle the dependency chaining, but munging file names when inducting them means that I have to unmunge them on every output. Suddenly you can't just zip or grep -H a selection of some of the files anymore; you have to do a whole dance of copy-and-unmunge-name, clean-up-the-temporary-directory, and so on and so on and. If I could just assign my own names, I could just as well avoid spaces, if it were purely an aesthetic issue. But it really isn't. You don't often need “something that displays as a space but you still get to decide its representation”, you need “passing through exactly the ASCII space without having to worry about it”, or the interoperability concerns still explode in your face one way or another (if they exist in the first place).
- dahart 9y ago> or you use special pattern rules I used make for years before I understood pattern rules, even though they're pretty simple. I kept trying to use foreach loops all over the place. If you're using foreach loops, check whether a pattern rule can do the job. > The popular clamour is that the autovars are too short & confusing. I finally realized, after years of using make and then learning pattern rules, that the three autovars listed here are visual mnemonics. $@ looks like a target, a ring with a bullseye. $< is pointing left, so it's the first prereq. $^ is like a horizontal bracket that groups all prereqs.
- henry_flower 9y agothat's a wonderful mnemonic you have! I've quoted you in the updated version of the text (I hope you don't mind).
- dahart 9y agoI don't mind at all, though credit should really be given to make's authors, I'm sure it was intentional.
- gkya 9y ago> I finally realized, after years of using make and then learning pattern rules, that the three autovars listed here are visual mnemonics. $@ looks like a target, a ring with a bullseye. $< is pointing left, so it's the first prereq. $^ is like a horizontal bracket that groups all prereqs. I've been illuminated, sincerely! Meanwhile, bmake has some more memorable synonyms for them like ".TARGET" and ".ALLSRC".
- sjmulder 9y agoIt's a shame that there is no name for .ALLSRC shared between bmake and gmake. An alternative is to use a variable, but then you lose VPATH compatibility: foo: foo.o bar.o $(LD) foo.o bar.o ... My other gripe with bmake is that this rule is needed as there is no implicit rule for linking multiple object files, or so it seems.
- 9y ago
- ejholmes 9y ago> Most important concept #1: DAG Yes! Thank you! This needs to be banged into peoples heads when talking about build systems. Every time you add an edge, you're automating an otherwise manual task. Very important realization.
- buserror 9y agoThe thing people miss a lot about makefile, apart from what has already been mentioned is the fact you can have "local" version of the "global" variables, per rules. It's very very powerfull, clear to read, and simplify all these if/else/end no end... For example: CFLAGS = -std=gnu99 -Werror release/%.elf: CFLAGS += -DRELEASE debug/%.elf: CFLAGS += -DDEBUG Basically you can add your own context flags and variables for each individual targets, and they are 'inherited' by any of their /sub/ targets... Oh, people also need to learn about VPATH or course [0]! [0]: https://www.gnu.org/software/make/manual/html_node/General-Search.html https://www.gnu.org/software/make/manual/html_node/General-S...
- henry_flower 9y agoyes, target-specific vars are neat. I've thought about adding a couple of sentences about vpath (in a cautious style akin to "think twice before using it" [0]), but ultimately decided that the topic is too obscure; besides I practically never use vpath myself. [0] http://make.mad-scientist.net/how-not-to-use-vpath/ http://make.mad-scientist.net/how-not-to-use-vpath/
- ghettoimp 9y agoThe writing is so awful. “Some ppl will get overexcited & start...”? “2fold”? Are we teenagers texting each other?
- abainbridge 9y agoI thought the same thing. However, I'd say the writing is mostly good. The short comings are mostly superficial.
- henry_flower 9y agothanks for the feedback. I've heard in the past about my "w/" and "&" misuse (apart from my pitiful grammar) but I haven't released it can put people off. I'm sorry.
- ghettoimp 9y agoSorry for being so harsh. I was in a terrible mood, but that's no excuse.
- henry_flower 9y agoit's alright besides, you're right, "ppl" and especially "2fold" were a bit too much :)