7 ms·

Self-Documented Makefile

249 points by fzaninotto 11y ago

jdp 11y ago
I like the ideas here, but for long-running processes like file watching, dev servers, hot reloading, etc. a better format is Procfile (https://devcenter.heroku.com/articles/procfile https://devcenter.heroku.com/articles/procfile). The ideas from this article could be nicely applied to it. Procfil is a format that declares a named list of processes to be run that can be controlled by tools like Foreman (http://ddollar.github.io/foreman/ http://ddollar.github.io/foreman/) and Honcho (https://pypi.python.org/pypi/honcho https://pypi.python.org/pypi/honcho). The advantage is being able to start and stop them concurrently as a group, useful for things that otherwise take a tmux session or multiple windows/tabs, like dev server + file watching + live reload: they become a simple `foreman start`. Processes can also be started individually. Procfiles can also be exported to other formats, like systemd, upstart, inittab, etc. Here's an example Procfile from a web project I've been working on. Since it uses node I went with node tools like http-server and watch, but it could just as easily use any other web server or file watcher. The way it works is it starts a web server serving public/; starts a live reload server for public/; and watches the src/ directory for changes and re-runs make. The makefile has a few rules for compiling JS and CSS from src/ to public/. web: ./node_modules/.bin/http-server livereload: ./node_modules/.bin/livereload public watch: ./node_modules/.bin/watch make src
- aikah 11y ago
  > but for long-running processes like file watching, dev servers, hot reloading, etc. I don't think anybody should use make to do that at first place. That's not what make was built for. Likewise Foreman should not be used as a build tool because it is not. EDIT: now i've seen the makefile in the example,I understand your comment and this is absolutely not where one wants to use make, that's just ridiculous.
  - solipsism 11y ago
    Wouldn't "it's not appropriate for the task" be a better reason not to use something than "it's not made for the task"? Don't you have any better reasons at all? Does make bring out people's conservative side or something? Let me ask you this. Would you sit on a tree stump? How about kill a fly with a newspaper? Sometimes things are great for purposes for which they weren't originally intended.
    - aikah 11y ago
      > Does make bring out people's conservative side or something? Misuse of tools in software development is why we end up with broken software, useless solutions that solve stupid problems because the problem wasn't well understood as first place, and first and foremost unnecessary dependencies. That's why we end up with this makefile "hack". Now explain what it's got to do with "conservatism". bad practices != innovation .
      - solipsism 11y ago
        Do you really believe any use of a tool in a way that wasn't intended is a "bad practice"? Is there no more subtlety or thought to it than that? This adherence to an ultra-simplistic black-and-white rule is absolutely a form of conservatism. If you think this particular use of make is a "bad practice", then argue why it is! If there's no better reason than "This use isn't as intended!" then your opinion won't have much weight with people.
  - jdp 11y ago
    The example isn't a makefile, it's a procfile.
    - aikah 11y ago
      I'm talking about the link, not the op.
- tempodox 11y ago
  An important point of the article was to use standard tools installed everywhere and not some obscure niche tools. Please note that you yourself felt the need to explain what “Procfil” is in the first place.
  - jdp 11y ago
    I wouldn't consider the industry backing of Heroku and a collective 4,500+ GitHub stars between the tools particularly niche or obscure. Crank it up to ~20k stars if you want to count deploy tools that can use them, like Dokku or Flynn. Anyway, the other great thing about Procfile is that it's just a declarative format. Lots of tools can leverage them, and they are an important part of my and many others' workflows. I also don't think the article is too keen on "standards", judging by it referring to make a "task launcher" and the suggested usage completely diverging from the expected behavior of the program.
    - qznc 11y ago
      Is it even packaged for Debian/Ubuntu yet? Make has been there for decades and is available on practically every developer machine out there. Compared to Make, nearly every tool is niche and obscure.
spoiler 11y ago
I've been using a similar self-documenting technique myself for a while now, too. Although, my version preserved the traditional part where just calling `make` starts building the program and also supported short one line descriptions and longer ones. Slightly OT: I like how Rake handles this, which is what gave me the idea in the first place
to3m 11y ago
You probably want to be using phony targets if your Makefile consists of stuff like this. See: https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html https://www.gnu.org/software/make/manual/html_node/Phony-Tar...
yxlx 11y ago
>Wouldn’t it better if we could just type make, and get a list of available commands, together with their desciption? No, Jesus Christ, please don't. Preserve default action as being to build. Good user interface and good user experience relies on meeting expectations. This behavior breaks those expectations. What if your expectations are different, you ask? In environments where there is an established tradition I think it is rude to break with the norm unless there is a compelling reason to do so. The commandline is popular among developers, other IT professionals and power users because of how efficient it is. It is efficient because there is not all the noise, handholding and other bullsh*t. Please let us keep it that way. Use a specific target to list other targets. I've seen some people use "make targets".
- andrewpe 11y ago
  I agree, don't break the normal expectations, but they could've done much much worst.
- gtramont 11y ago
  This is basically how I've been doing for quite some time now. `make ?` Check it out here: https://github.com/gtramontina/sourcing/blob/master/Makefile#L68-L72 https://github.com/gtramontina/sourcing/blob/master/Makefile...
  - jjnoakes 11y ago
    If your users set failglob in their shells, they will have to type the less convenient "make '?'" or "make \?". And regardless of whether they do or don't, if they are in a directory with a single character file name, they will have to escape it too...
    - gtramont 11y ago
      I usually escape regardless... But you made a good point about the single character directory. The target name can be anything that makes sense, anyway... `make help` as suggested previously.
- 11y ago
chubot 11y ago
They should really be using bash for this, not make. There is nothing wrong with bash scripts calling Make -- for building with DEPENDENCIES. But when you aren't doing that, just use bash (because Make is actually Make + bash to begin with). This is dumb: restart-frontend: ## Restart the frontend and admin apps in dev @make stop-frontend-dev && make run-frontend-dev @echo "Frontend app restarted" Write it with a shell script like this: stop-frontend-dev() { ... } run-frontend-dev() { ... } restart-front-end() { stop-frontend-dev run-frontend-dev echo "Frontend app restarted" } build() { make # this actually does stuff you can't do in bash. } "$@" # call function $1 with args $2... Can also print help here. This is a lot cleaner. The PID stuff can be done with bash too.
- accatyyc 11y ago
  But why does it matter? With your solution they'll have two "task files" to manage instead of one.
  - antocv 11y ago
    Because make is for managing dependencies, which it tracks by files, as soon as you need to do something which doesnt leave files around, such as start a new service, you then need to plug this hole, by using "touch" and leaving files around, which this example doesnt do so it may actually leave the system inconsistent and make the makefile do things it shouldnt, thus causing errors difficult to diagnose. But then you run into the problem of making make have functions of a service manager, which it isnt, and you will most likely fail doing that. Use Make when you need to track dependencies based on files, such as for build systems whose output is files, and bash/other script for managing a hosts services which are/may be required for the build system.
- xaduha 11y ago
  Nope, to me it isn't cleaner. Also, make has "make -n" which is absolutely golden.
  - 11y ago
jgrahamc 11y ago
An alternative way that just uses GNU make functions: http://www.cmcrossroads.com/article/self-documenting-makefiles http://www.cmcrossroads.com/article/self-documenting-makefil...
tempodox 11y ago
Change `grep -P` to `grep -E` (or simply `egrep`) and it also works on OS X.
- fzaninotto 11y ago
  Yep, that fixes the OSX problem. Post updated with this version.
m6w6 11y ago
Shorter, more fail-proof help target: help: @awk -F ':|##' \ '/^[^\t].+?:.*?##/ {\ printf "\033[36m%-30s\033[0m %s\n", $$1, $$NF \ }' $(MAKEFILE_LIST) ... but I agree, breaking expectations is somewhat bad. Also, many shells have completion for Makefiles nowadays, though, that won't get you an additional help text.
kluck 11y ago
Good idea but target names might contain numbers as well, so you should adjust the regular expression used: @grep -P '^[a-zA-Z0-9_-]+:.?## .$$' $(MAKEFILE_LIST) | ...
beaufils 11y ago
Here is a version with no dependencies to grep or awk but just sed. I did not tested it on OS X yet. help: @eval $$(sed -r -n 's/^([a-zA-Z0-9_-]+):.*?## (.*)$$/printf "\\033[36m%-30s\\033[0m %s\\n" "\1" "\2" ;/; ta; b; :a p' $(MAKEFILE_LIST) | sort)
- LukeShu 11y ago
  I don't have an OS X box, but I do know that you'll need to change the `-r` to `-E` (GNU sed vs BSD sed). Recent versions of GNU sed (4.2+, I think) also accept `-E` for compatibility with BSD sed (though this is undocumented).
rekado 11y ago
Instead of using `awk` to break lines you could just use `fmt`, which is part of the GNU coreutils.
ehartsuyker 11y ago
I do this in my bash scripts, if anyone wants to see how. https://github.com/ehartsuyker/node-deb/blob/ede596b2c8a07f1f1925e4f8e51df5ba9ef15be3/node-deb#L44 https://github.com/ehartsuyker/node-deb/blob/ede596b2c8a07f1...
Gratsby 11y ago
But then you have lost the default target of make and instead of make && make install you end up with make build && make install. That's going to break a brain or two when people try to figure out why their default MO doesn't work.
saiki 11y ago
Thanks for sharing! This is a great way to document and see documentation for main targets in Makefile. We use Makefile in same way to execute project related tasks like deployments and run development environments. This will even further help to show main targets from a Makefile easily and pretty standard way. Will be taken into use. You can achieve similar by writing bash scripts, but it will be mostly custom and others need to learn how to use it and extend it. Makefile gives you a standard way of writing small utilities related to all your project, and almost everybody knows how Makefile works and if not, they can learn from existing documentation.
ArnaudRinquin 11y ago
As I used `make` more and more for our node projects, I missed the clean outputs `grunt` or `gulp` provide. To fix that, I created `make2tap`: https://www.npmjs.com/package/make2tap https://www.npmjs.com/package/make2tap This small utility takes a `make` output and generate a `TAP` one that you can pipe to any `TAP reporter`. Our current `make build | make2tap | tap-format-spec` looks like: http://i.imgur.com/chs0Jf3.png http://i.imgur.com/chs0Jf3.png
- agumonkey 11y ago
  Beautiful, just beautiful.

249 points by fzaninotto 11y ago

jdp 11y ago

I like the ideas here, but for long-running processes like file watching, dev servers, hot reloading, etc. a better format is Procfile (https://devcenter.heroku.com/articles/procfile https://devcenter.heroku.com/articles/procfile). The ideas from this article could be nicely applied to it. Procfil is a format that declares a named list of processes to be run that can be controlled by tools like Foreman (http://ddollar.github.io/foreman/ http://ddollar.github.io/foreman/) and Honcho (https://pypi.python.org/pypi/honcho https://pypi.python.org/pypi/honcho). The advantage is being able to start and stop them concurrently as a group, useful for things that otherwise take a tmux session or multiple windows/tabs, like dev server + file watching + live reload: they become a simple `foreman start`. Processes can also be started individually. Procfiles can also be exported to other formats, like systemd, upstart, inittab, etc. Here's an example Procfile from a web project I've been working on. Since it uses node I went with node tools like http-server and watch, but it could just as easily use any other web server or file watcher. The way it works is it starts a web server serving public/; starts a live reload server for public/; and watches the src/ directory for changes and re-runs make. The makefile has a few rules for compiling JS and CSS from src/ to public/. web: ./node_modules/.bin/http-server livereload: ./node_modules/.bin/livereload public watch: ./node_modules/.bin/watch make src

aikah 11y ago
> but for long-running processes like file watching, dev servers, hot reloading, etc. I don't think anybody should use make to do that at first place. That's not what make was built for. Likewise Foreman should not be used as a build tool because it is not. EDIT: now i've seen the makefile in the example,I understand your comment and this is absolutely not where one wants to use make, that's just ridiculous.
- solipsism 11y ago
  Wouldn't "it's not appropriate for the task" be a better reason not to use something than "it's not made for the task"? Don't you have any better reasons at all? Does make bring out people's conservative side or something? Let me ask you this. Would you sit on a tree stump? How about kill a fly with a newspaper? Sometimes things are great for purposes for which they weren't originally intended.
  - aikah 11y ago
    > Does make bring out people's conservative side or something? Misuse of tools in software development is why we end up with broken software, useless solutions that solve stupid problems because the problem wasn't well understood as first place, and first and foremost unnecessary dependencies. That's why we end up with this makefile "hack". Now explain what it's got to do with "conservatism". bad practices != innovation .
    - solipsism 11y ago
      Do you really believe any use of a tool in a way that wasn't intended is a "bad practice"? Is there no more subtlety or thought to it than that? This adherence to an ultra-simplistic black-and-white rule is absolutely a form of conservatism. If you think this particular use of make is a "bad practice", then argue why it is! If there's no better reason than "This use isn't as intended!" then your opinion won't have much weight with people.
- jdp 11y ago
  The example isn't a makefile, it's a procfile.
  - aikah 11y ago
    I'm talking about the link, not the op.
tempodox 11y ago
An important point of the article was to use standard tools installed everywhere and not some obscure niche tools. Please note that you yourself felt the need to explain what “Procfil” is in the first place.
- jdp 11y ago
  I wouldn't consider the industry backing of Heroku and a collective 4,500+ GitHub stars between the tools particularly niche or obscure. Crank it up to ~20k stars if you want to count deploy tools that can use them, like Dokku or Flynn. Anyway, the other great thing about Procfile is that it's just a declarative format. Lots of tools can leverage them, and they are an important part of my and many others' workflows. I also don't think the article is too keen on "standards", judging by it referring to make a "task launcher" and the suggested usage completely diverging from the expected behavior of the program.
  - qznc 11y ago
    Is it even packaged for Debian/Ubuntu yet? Make has been there for decades and is available on practically every developer machine out there. Compared to Make, nearly every tool is niche and obscure.

spoiler 11y ago

I've been using a similar self-documenting technique myself for a while now, too. Although, my version preserved the traditional part where just calling `make` starts building the program and also supported short one line descriptions and longer ones. Slightly OT: I like how Rake handles this, which is what gave me the idea in the first place

to3m 11y ago

You probably want to be using phony targets if your Makefile consists of stuff like this. See: https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html https://www.gnu.org/software/make/manual/html_node/Phony-Tar...

yxlx 11y ago

>Wouldn’t it better if we could just type make, and get a list of available commands, together with their desciption? No, Jesus Christ, please don't. Preserve default action as being to build. Good user interface and good user experience relies on meeting expectations. This behavior breaks those expectations. What if your expectations are different, you ask? In environments where there is an established tradition I think it is rude to break with the norm unless there is a compelling reason to do so. The commandline is popular among developers, other IT professionals and power users because of how efficient it is. It is efficient because there is not all the noise, handholding and other bullsh*t. Please let us keep it that way. Use a specific target to list other targets. I've seen some people use "make targets".

andrewpe 11y ago
I agree, don't break the normal expectations, but they could've done much much worst.
gtramont 11y ago
This is basically how I've been doing for quite some time now. `make ?` Check it out here: https://github.com/gtramontina/sourcing/blob/master/Makefile#L68-L72 https://github.com/gtramontina/sourcing/blob/master/Makefile...
- jjnoakes 11y ago
  If your users set failglob in their shells, they will have to type the less convenient "make '?'" or "make \?". And regardless of whether they do or don't, if they are in a directory with a single character file name, they will have to escape it too...
  - gtramont 11y ago
    I usually escape regardless... But you made a good point about the single character directory. The target name can be anything that makes sense, anyway... `make help` as suggested previously.
11y ago

chubot 11y ago

They should really be using bash for this, not make. There is nothing wrong with bash scripts calling Make -- for building with DEPENDENCIES. But when you aren't doing that, just use bash (because Make is actually Make + bash to begin with). This is dumb: restart-frontend: ## Restart the frontend and admin apps in dev @make stop-frontend-dev && make run-frontend-dev @echo "Frontend app restarted" Write it with a shell script like this: stop-frontend-dev() { ... } run-frontend-dev() { ... } restart-front-end() { stop-frontend-dev run-frontend-dev echo "Frontend app restarted" } build() { make # this actually does stuff you can't do in bash. } "$@" # call function $1 with args $2... Can also print help here. This is a lot cleaner. The PID stuff can be done with bash too.

accatyyc 11y ago
But why does it matter? With your solution they'll have two "task files" to manage instead of one.
- antocv 11y ago
  Because make is for managing dependencies, which it tracks by files, as soon as you need to do something which doesnt leave files around, such as start a new service, you then need to plug this hole, by using "touch" and leaving files around, which this example doesnt do so it may actually leave the system inconsistent and make the makefile do things it shouldnt, thus causing errors difficult to diagnose. But then you run into the problem of making make have functions of a service manager, which it isnt, and you will most likely fail doing that. Use Make when you need to track dependencies based on files, such as for build systems whose output is files, and bash/other script for managing a hosts services which are/may be required for the build system.
xaduha 11y ago
Nope, to me it isn't cleaner. Also, make has "make -n" which is absolutely golden.
- 11y ago

jgrahamc 11y ago

An alternative way that just uses GNU make functions: http://www.cmcrossroads.com/article/self-documenting-makefiles http://www.cmcrossroads.com/article/self-documenting-makefil...

tempodox 11y ago

Change `grep -P` to `grep -E` (or simply `egrep`) and it also works on OS X.

fzaninotto 11y ago
Yep, that fixes the OSX problem. Post updated with this version.

m6w6 11y ago

Shorter, more fail-proof help target: help: @awk -F ':|##' \ '/^[^\t].+?:.*?##/ {\ printf "\033[36m%-30s\033[0m %s\n", $$1, $$NF \ }' $(MAKEFILE_LIST) ... but I agree, breaking expectations is somewhat bad. Also, many shells have completion for Makefiles nowadays, though, that won't get you an additional help text.

kluck 11y ago

Good idea but target names might contain numbers as well, so you should adjust the regular expression used: @grep -P '^[a-zA-Z0-9_-]+:.?## .$$' $(MAKEFILE_LIST) | ...

beaufils 11y ago

Here is a version with no dependencies to grep or awk but just sed. I did not tested it on OS X yet. help: @eval $$(sed -r -n 's/^([a-zA-Z0-9_-]+):.*?## (.*)$$/printf "\\033[36m%-30s\\033[0m %s\\n" "\1" "\2" ;/; ta; b; :a p' $(MAKEFILE_LIST) | sort)

LukeShu 11y ago
I don't have an OS X box, but I do know that you'll need to change the `-r` to `-E` (GNU sed vs BSD sed). Recent versions of GNU sed (4.2+, I think) also accept `-E` for compatibility with BSD sed (though this is undocumented).

rekado 11y ago

Instead of using `awk` to break lines you could just use `fmt`, which is part of the GNU coreutils.

ehartsuyker 11y ago

I do this in my bash scripts, if anyone wants to see how. https://github.com/ehartsuyker/node-deb/blob/ede596b2c8a07f1f1925e4f8e51df5ba9ef15be3/node-deb#L44 https://github.com/ehartsuyker/node-deb/blob/ede596b2c8a07f1...

Gratsby 11y ago

But then you have lost the default target of make and instead of make && make install you end up with make build && make install. That's going to break a brain or two when people try to figure out why their default MO doesn't work.

saiki 11y ago

Thanks for sharing! This is a great way to document and see documentation for main targets in Makefile. We use Makefile in same way to execute project related tasks like deployments and run development environments. This will even further help to show main targets from a Makefile easily and pretty standard way. Will be taken into use. You can achieve similar by writing bash scripts, but it will be mostly custom and others need to learn how to use it and extend it. Makefile gives you a standard way of writing small utilities related to all your project, and almost everybody knows how Makefile works and if not, they can learn from existing documentation.

ArnaudRinquin 11y ago

As I used `make` more and more for our node projects, I missed the clean outputs `grunt` or `gulp` provide. To fix that, I created `make2tap`: https://www.npmjs.com/package/make2tap https://www.npmjs.com/package/make2tap This small utility takes a `make` output and generate a `TAP` one that you can pipe to any `TAP reporter`. Our current `make build | make2tap | tap-format-spec` looks like: http://i.imgur.com/chs0Jf3.png http://i.imgur.com/chs0Jf3.png

agumonkey 11y ago
Beautiful, just beautiful.