6 ms·

Rebuild code automatically with "watch" and a makefile

42 points by graue 14y ago

jkrippy 14y ago
I use 'entr' for this, which uses kqueue(2) to avoid polling: * http://entrproject.org/ * https://bitbucket.org/eradman/entr/
- graue 14y ago
  Neat, that looks like a much more well-engineered and less hacky solution. While I'm happy with what I have now, I've bookmarked this in case I need it later for a bigger/more complicated project.
  - mitchty 14y ago
    I like it, nice and simple too, not even a configure script. Created a tap for it on homebrew if anyone wants to use it. brew tap mitchty/entr if you want to test it out.
- norswap 14y ago
  There is also watchr: https://github.com/mynyml/watchr https://github.com/mynyml/watchr
emillon 14y ago
On linux you can use inotify to avoid polling. I use such a rule for my makefiles : watch: while true ; do inotifywait -qe close_write $(SRC); $(MAKE) ; done
- sp332 14y ago
  How do you use it? Like: $ make watch & $ vim Or do you use separate screens?
  - emillon 14y ago
    I use separate screens. Usually I have one terminal for vim and one terminal with a shell for make, git, etc. Using this setup and a tiling window manager, I can see the build log whenever I type :w.
- Hello71 14y ago
  while inotifywait -qe close_write $(SRC); do $(MAKE); done
  - emillon 14y ago
    This will stop trying to build at the first compile error.
    - Hello71 14y ago
      No, it will stop when the directory disappears.
npsimons 14y ago
Hah! I love it! Continuous integration with two UNIX utilities!
Zuph 14y ago
Or you can use a build system like Tup, which uses inotify to notice when any dependencies change: http://gittup.org/tup/ http://gittup.org/tup/
joshka 14y ago
A node version of this is to use supervisor [1]. E.g. in a makefile [2] for http://countw.tf/ http://countw.tf/ (a great way to vent while exploring an old codebase with many err... sub-optimal choices) debug: browserify site/scripts/app.js -o site/scripts/app-built.js --debug watch: supervisor --watch site/scripts --ignore site/scripts/app-built.js --no-restart-on exit --exec make debug 1: https://github.com/isaacs/node-supervisor https://github.com/isaacs/node-supervisor 2: https://github.com/joshka/countw.tf/blob/master/Makefile https://github.com/joshka/countw.tf/blob/master/Makefile
sigil 14y ago
The use of watch seems pretty pointless here, since you specifically don't want the original watch(1) behavior of clearing the terminal and redrawing. Equivalent and no special tool required: while true; do make all; sleep 1; done One problem that becomes apparent with a longer interval and longer rebuild times: if the make takes say 30s, and you're rebuilding every 60s, then your site will be updated every 1.5 minutes instead of every minute. Maybe not what you want. A program you should really check out is trigger-listen [1]. It can run a command every minute regardless of how long the command takes to complete (as long as it takes < 1min). You can also trigger a rebuild whenever you want by writing a byte into a fifo. I use it to rebuild static sites when a certain branch is updated -- just do a trigger-pull in your git post-receive hook. [1] http://www.superscript.com/trigger/trigger-listen.html http://www.superscript.com/trigger/trigger-listen.html
- 1amzave 14y ago
  > One problem that becomes apparent with a longer interval and longer rebuild times: if the make takes say 30s, and you're rebuilding every 60s, then your site will be updated every 1.5 minutes instead of every minute. Maybe not what you want. Easy enough to fix without resorting to a specialized tool: while true; do make all & sleep 1m; done (Changing the sleep to a minute instead of a second, which may have been what you intended.) That way you'll kick off a 'make all' once per minute regardless of how long it takes. That in turn might not be what you want if your make takes more than a minute, but then it's also simple enough to extend it thusly: while true; do make all & sleep 1m & wait; done which will do the same but avoid having multiple makes running concurrently (if a minute elapses and the previous invocation hasn't finished yet, it waits for it before starting the next one).
  - sigil 14y ago
    while true; do make all & sleep 1m; done Welcome to orphan land. (^C to see.) Don't do this folks. The 'make all' invocations may also potentially overlap and corrupt your builds. To be fair, you fixed that issue here: while true; do make all & sleep 1m & wait; done But the orphan issue remains.
jvdh 14y ago
I've used something similar before, but I've always found that after a while it makes me wary to save a file. Saving means a rebuild, meaning potential errors, leading to a let-down or sense of failure. While computers have become a lot more stable, I still like to save often, and this kind of setup eventually makes you feel less like you want to save.
- graue 14y ago
  Interesting point. I usually don't mind breaking the build for a few minutes while implementing a biggish change, but I also commit often and can use "git diff" to find out what I changed and quickly get back to a working state.
i_s 14y ago
If you're on mac, you can use fswatch instead: https://github.com/alandipert/fswatch https://github.com/alandipert/fswatch It uses Mac APIs to monitor the file system and executes your command when it changes.
- stefans 14y ago
  And there is https://bitbucket.org/ssaasen/spy https://bitbucket.org/ssaasen/spy if you want something that works on both Mac and Linux (disclaimer: author here). The "run" mode can be used for the "compile on fs event" use case.
philsnow 14y ago
I suppose I should write up this ancient use of inotifywait and xpdf's -remote option on a blog somewhere: https://news.ycombinator.com/item?id=5276477 https://news.ycombinator.com/item?id=5276477
eeperson 14y ago
The ability to do this based on file modifications is my favorite feature of the build tool SBT (a Java/Scala build tool). I really wish more build tools would integrate this natively. Usually this functionality only shows up as a plugin for a specific build task (such a Spork for Rails testing) or as a platform specific command line hack (which isn't as useful if the build tool has a large start up time).
andrewf 14y ago
My watch-make tool tries to do this "properly"; it uses GNU make's --print-data-base to figure out which files to watch for a change. You can run "watch-make test4" and it'll only run "make test4" when a file which is in the dependency chain for test4 changes, rather than any file in the directory tree. It tweaks the PATH for child processes so recursive make invocations should work. I've been lazy and not put yet made an npm package (it's implemented in node.js) or tested cross-platform. Guess I've no reason to be bored tomorrow. https://github.com/andrewffff/watch-make https://github.com/andrewffff/watch-make
- graue 14y ago
  Awesome, starred it. I hope you do put it on npm!
glenjamin 14y ago
For a nice, portable and generic version of the "do something when the filesystem changes" flavour of tool, check out the "watchmedo" command provided by watchdog (https://pypi.python.org/pypi/watchdog https://pypi.python.org/pypi/watchdog). It's quite similar to watchr in ruby-land, but the executable is easier to use without creating a config file.
kevinburke 14y ago
If you're interested in this area there's a Ruby project which does the correct thing here, which is to only recompile/run `make` when a file changes. The original author has stopped maintaining the project. I've been working on a fork, called `observr`, and looking for contributors: https://github.com/kevinburke/observr https://github.com/kevinburke/observr
MikeCodeAwesome 14y ago
Yet another fun use of watch, one of my favorite tools! I use it daily in combination with curl and grep to look for certain output of my JSON/XML services during development. The -d flag is especially useful in this regard. watch -d -n 5 "curl -s localhost:8080/some/service | grep status" Along the same lines as the original post, I also use watch as an ad hoc, pseudo continuous integration tool for executing code that doesn't fit well within a REPL. [1] [1] http://codeaweso.me/2012/09/using-watch-as-quick-continuous-integration/ http://codeaweso.me/2012/09/using-watch-as-quick-continuous-...
Tarential 14y ago
I've done something similar with inotify for compiling/concatenating CoffeeScript/SCSS/Haml in case anyone wants to use it: https://github.com/Tarential/autobuild-client https://github.com/Tarential/autobuild-client

42 points by graue 14y ago

I use 'entr' for this, which uses kqueue(2) to avoid polling: * http://entrproject.org/ * https://bitbucket.org/eradman/entr/

graue 14y ago
Neat, that looks like a much more well-engineered and less hacky solution. While I'm happy with what I have now, I've bookmarked this in case I need it later for a bigger/more complicated project.
- mitchty 14y ago
  I like it, nice and simple too, not even a configure script. Created a tap for it on homebrew if anyone wants to use it. brew tap mitchty/entr if you want to test it out.
norswap 14y ago
There is also watchr: https://github.com/mynyml/watchr https://github.com/mynyml/watchr

emillon 14y ago

On linux you can use inotify to avoid polling. I use such a rule for my makefiles : watch: while true ; do inotifywait -qe close_write $(SRC); $(MAKE) ; done

sp332 14y ago
How do you use it? Like: $ make watch & $ vim Or do you use separate screens?
- emillon 14y ago
  I use separate screens. Usually I have one terminal for vim and one terminal with a shell for make, git, etc. Using this setup and a tiling window manager, I can see the build log whenever I type :w.
Hello71 14y ago
while inotifywait -qe close_write $(SRC); do $(MAKE); done
- emillon 14y ago
  This will stop trying to build at the first compile error.
  - Hello71 14y ago
    No, it will stop when the directory disappears.

npsimons 14y ago

Hah! I love it! Continuous integration with two UNIX utilities!

Zuph 14y ago

Or you can use a build system like Tup, which uses inotify to notice when any dependencies change: http://gittup.org/tup/ http://gittup.org/tup/

joshka 14y ago

A node version of this is to use supervisor [1]. E.g. in a makefile [2] for http://countw.tf/ http://countw.tf/ (a great way to vent while exploring an old codebase with many err... sub-optimal choices) debug: browserify site/scripts/app.js -o site/scripts/app-built.js --debug watch: supervisor --watch site/scripts --ignore site/scripts/app-built.js --no-restart-on exit --exec make debug 1: https://github.com/isaacs/node-supervisor https://github.com/isaacs/node-supervisor 2: https://github.com/joshka/countw.tf/blob/master/Makefile https://github.com/joshka/countw.tf/blob/master/Makefile

sigil 14y ago

The use of watch seems pretty pointless here, since you specifically don't want the original watch(1) behavior of clearing the terminal and redrawing. Equivalent and no special tool required: while true; do make all; sleep 1; done One problem that becomes apparent with a longer interval and longer rebuild times: if the make takes say 30s, and you're rebuilding every 60s, then your site will be updated every 1.5 minutes instead of every minute. Maybe not what you want. A program you should really check out is trigger-listen [1]. It can run a command every minute regardless of how long the command takes to complete (as long as it takes < 1min). You can also trigger a rebuild whenever you want by writing a byte into a fifo. I use it to rebuild static sites when a certain branch is updated -- just do a trigger-pull in your git post-receive hook. [1] http://www.superscript.com/trigger/trigger-listen.html http://www.superscript.com/trigger/trigger-listen.html

1amzave 14y ago
> One problem that becomes apparent with a longer interval and longer rebuild times: if the make takes say 30s, and you're rebuilding every 60s, then your site will be updated every 1.5 minutes instead of every minute. Maybe not what you want. Easy enough to fix without resorting to a specialized tool: while true; do make all & sleep 1m; done (Changing the sleep to a minute instead of a second, which may have been what you intended.) That way you'll kick off a 'make all' once per minute regardless of how long it takes. That in turn might not be what you want if your make takes more than a minute, but then it's also simple enough to extend it thusly: while true; do make all & sleep 1m & wait; done which will do the same but avoid having multiple makes running concurrently (if a minute elapses and the previous invocation hasn't finished yet, it waits for it before starting the next one).
- sigil 14y ago
  while true; do make all & sleep 1m; done Welcome to orphan land. (^C to see.) Don't do this folks. The 'make all' invocations may also potentially overlap and corrupt your builds. To be fair, you fixed that issue here: while true; do make all & sleep 1m & wait; done But the orphan issue remains.

jvdh 14y ago

I've used something similar before, but I've always found that after a while it makes me wary to save a file. Saving means a rebuild, meaning potential errors, leading to a let-down or sense of failure. While computers have become a lot more stable, I still like to save often, and this kind of setup eventually makes you feel less like you want to save.

graue 14y ago
Interesting point. I usually don't mind breaking the build for a few minutes while implementing a biggish change, but I also commit often and can use "git diff" to find out what I changed and quickly get back to a working state.

i_s 14y ago

If you're on mac, you can use fswatch instead: https://github.com/alandipert/fswatch https://github.com/alandipert/fswatch It uses Mac APIs to monitor the file system and executes your command when it changes.

stefans 14y ago
And there is https://bitbucket.org/ssaasen/spy https://bitbucket.org/ssaasen/spy if you want something that works on both Mac and Linux (disclaimer: author here). The "run" mode can be used for the "compile on fs event" use case.

philsnow 14y ago

I suppose I should write up this ancient use of inotifywait and xpdf's -remote option on a blog somewhere: https://news.ycombinator.com/item?id=5276477 https://news.ycombinator.com/item?id=5276477

eeperson 14y ago

The ability to do this based on file modifications is my favorite feature of the build tool SBT (a Java/Scala build tool). I really wish more build tools would integrate this natively. Usually this functionality only shows up as a plugin for a specific build task (such a Spork for Rails testing) or as a platform specific command line hack (which isn't as useful if the build tool has a large start up time).

andrewf 14y ago

My watch-make tool tries to do this "properly"; it uses GNU make's --print-data-base to figure out which files to watch for a change. You can run "watch-make test4" and it'll only run "make test4" when a file which is in the dependency chain for test4 changes, rather than any file in the directory tree. It tweaks the PATH for child processes so recursive make invocations should work. I've been lazy and not put yet made an npm package (it's implemented in node.js) or tested cross-platform. Guess I've no reason to be bored tomorrow. https://github.com/andrewffff/watch-make https://github.com/andrewffff/watch-make

graue 14y ago
Awesome, starred it. I hope you do put it on npm!

glenjamin 14y ago

For a nice, portable and generic version of the "do something when the filesystem changes" flavour of tool, check out the "watchmedo" command provided by watchdog (https://pypi.python.org/pypi/watchdog https://pypi.python.org/pypi/watchdog). It's quite similar to watchr in ruby-land, but the executable is easier to use without creating a config file.

kevinburke 14y ago

If you're interested in this area there's a Ruby project which does the correct thing here, which is to only recompile/run `make` when a file changes. The original author has stopped maintaining the project. I've been working on a fork, called `observr`, and looking for contributors: https://github.com/kevinburke/observr https://github.com/kevinburke/observr

MikeCodeAwesome 14y ago

Yet another fun use of watch, one of my favorite tools! I use it daily in combination with curl and grep to look for certain output of my JSON/XML services during development. The -d flag is especially useful in this regard. watch -d -n 5 "curl -s localhost:8080/some/service | grep status" Along the same lines as the original post, I also use watch as an ad hoc, pseudo continuous integration tool for executing code that doesn't fit well within a REPL. [1] [1] http://codeaweso.me/2012/09/using-watch-as-quick-continuous-integration/ http://codeaweso.me/2012/09/using-watch-as-quick-continuous-...

Tarential 14y ago

I've done something similar with inotify for compiling/concatenating CoffeeScript/SCSS/Haml in case anyone wants to use it: https://github.com/Tarential/autobuild-client https://github.com/Tarential/autobuild-client