7 ms·

Watchman: Execute a command when something changes

248 points by ColinWright 4y ago

stoplying1 4y ago
Watchman works great, but the CLI UX is... really kinda dreadful in my opinion. Trying to get it usable in diverse scenarios kinda reveals that the author has a very particular way it's meant to be used and it seems needlessly frustrating to use it otherwise.
BrainVirus 4y ago
This command seems like a no-brainer that should be in all OSes by default. It's mind-boggling that most systems don't have a standard tool for observing file changes. The only language I know of that has a platform-agnostic API for this is Factor.
- dspillett 4y ago
  The required kernel hooks exist in pretty much any common OS these days, it is a user-space tool that is sometimes missing. It may not be installed by default, but inotifywait is available in common Linux distributions, usually in a package called something like ionotify-tools, and has been for over a decade-ana-half IIRC. It'll work under WSL on Windows too, though only for ext4 devices not bits of the Windows filesystem made available to Linux. I can't speak to what other OSs include by default, but as every major OS has a different API for how to register a listener & how it gets messages no built-in tool is going to be cross-platform. There are third party tools which present more cross-platform consistency, most notably https://github.com/emcrisostomo/fswatch#readme https://github.com/emcrisostomo/fswatch#readme (also available in common Linux distros, just an apt install away in Debian for instance).
jitl 4y ago
Not to be confused with Facebook’s file watch daemon, also names watchman, which does the same sort of thing but is more complicated. There’s a bunch of tools that integrate Facebook’s watchman for more efficient change tracking. Advantages of Facebook’s watchman: 1. Implements efficient file system event watches on macOS and Windows 2. IPC/daemon system reduces resource use because overlapping watches/triggers don’t use more inotify slots. 3. Denounces / waits for changes to settle 4. Client libraries in a few different languages for scripting purposes https://facebook.github.io/watchman/ https://facebook.github.io/watchman/
- HelloNurse 4y ago
  5. Windows is supported.
- deleted 4y ago
  [deleted]
- dec0dedab0de 4y ago
  Also not to be confused with watchdog and it's command watchmedo, which is what I thought this was when I first read the headline. I don't know what the differences are, but I use watchdog to restart my local celery workers when I make changes. It's scripted out so I don't ever use the command directly. https://pypi.org/project/watchdog/ https://pypi.org/project/watchdog/ https://github.com/gorakhargosh/watchdog https://github.com/gorakhargosh/watchdog EDIT: Now I'm wondering how often I miss out on something because I confuse it for something I already use. Maybe there is room for a "things I use" or "things I know about" browser plugin.
  - ghassanmas 4y ago
    Also to note "watchdog" the term. Is used also in hardware/software context. Usually as a way to mitgate or recover system faulate. I first I heard of this term while taking in a course about PIC microcontroller[1]. > The Watchdog Timer in the PIC16F819 The watchdog timer can be a real source of pain and can also make a PIC system incredibly robust and reliable. But what exactly is the watchdog timer? Simply put, the watchdog timer is a hardware timer in the PIC that, if not constantly reset by the software, will cause the PIC to reset. This feature can be incredibly useful if the PIC should hang due to a hardware or software issue and guarantees that the PIC will restart from the beginning. Not only does it reset the system, it also flags a bit that can be used to determine if the system just crashed. Ref: https://maker.pro/pic/tutorial/how-to-get-started-with-pic-microcontrollers-timer0-and-the-watchdog-timer https://maker.pro/pic/tutorial/how-to-get-started-with-pic-m... [1]: PIC microcontroller, is a small hardware simliar to arduino but more on a lower level.
  - damagednoob 4y ago
    Also not to be confused with watchexec (my tool of choice for executing commands on file changes): https://github.com/watchexec/watchexec https://github.com/watchexec/watchexec
GRBurst 4y ago
For those use-cases I use entr (https://github.com/clibs/entr https://github.com/clibs/entr). In what sense does watchman differ to it? Didn't see anything related to this at first glance :-)
- deleted 4y ago
  [deleted]
- everybodyknows 4y ago
  The page linked above says: > WARNING: This is a (possibly outdated and/or unmaintained) fork of https://github.com/eradman/entr https://github.com/eradman/entr .
  - deleted 4y ago
    [deleted]
- DotaFan 4y ago
  OP did say he did this for learning purpose.
basemi 4y ago
I've used `incron`[0] for that purpose, with success. Packaged for most distros I think [0]https://github.com/ar-/incron https://github.com/ar-/incron
IncRnd 4y ago
What does this add to inotifywait? That does exactly the same as this project, which uses inotifywait to reimplement inotifywait. For example: dnf install inotify-tools or apt-get install inotify-tools
- masukomi 4y ago
  ifnotifywait doesn't exist on macOS, so in addition to that not being an option on macos, this would be something you could write tools that a team who used multiple platforms, or a person with setups on multiple os's, could use. I suspect this might even be usable on windows. [edit: fswatch looks very promising, and may be a better choice than either ifnotifywait or watchman]
  - IncRnd 4y ago
    You are confused about how the tool works. This project uses inotifywait from inotify tools to reimplement what inotifywait already does. The linked article page at github clearly says this. Adding a new layer atop inotify tools doesn't make that package automatically run on new platforms. There are already tools on macos that use the native FSEvents API for watching for file changes and executing scripts.
  - awinder 4y ago
    This wraps inotifywait, so it doesn’t really address cross-platform needs. The unique features in this over straight inotifywait is plugging together the command you want to execute, and some event dedupe logic. Grand majority of the code is around colorized logs, command docs, and some options handling code which mostly just gets piped to inotifywait.
  - wwweston 4y ago
    Discovered fswatch recently, it's proved very useful, my favorite use being replacing a few situations where I was using sshfs with fswatch+rsync: no network lag on save/load, can even work offline, but everything still makes its way to the network destination!
    - 4y ago
majodev 4y ago
Very happy user of https://github.com/watchexec/watchexec https://github.com/watchexec/watchexec (rust), can recommend.
- loevborg 4y ago
  Yes, this is the one I reach for always. It works reliably AFAICT, runs on all platforms, is simple enough and still has the necessary features.
- maximilianroos 4y ago
  Very much agree. And the maintainer has been super responsive.
- crodjer 4y ago
  This looks great! I'll also check this out as someone who pivoted to rust tooling in the recent years.
amelius 4y ago
From my experience, these types of notification handlers are very fragile and tend to fail every so often. Do not use if your life depends on it.
- remram 4y ago
  From my experience, they break every time if you are making changes with tools that replace the file rather than change it in-place, like most text editors.
  - amelius 4y ago
    In that case, you might want to monitor the directory too, not just the file. I believe this is even how it is supposed to work.
    - remram 4y ago
      Yeah, and then filter out the events in the directory to find out when the correct file changed, etc. That is exactly what I would want the tool to abstract for me.
  - Brian_K_White 4y ago
    At least with inotify, ie, incron, this is just a matter of setting a less stupid watch. You can ask the kernel to watch for several different specific kinds of fs events, either at the file or directory level. Detecting when a file is either created, or closed from writing, is no problem at all. Ie, handles the case of writing a file normally, editing a file, and renaming a file into exitence without having opened it for writing (that happened while it had some other name you weren't watching). This would handle unlinking and recreating the same as editing or uploading. You can watch specifically for the close event only, and even more specifically for close-from-write-mode to detect updates but ignore until the update is done. And at the same time you can also watch for creation, which handles creating a file under some other name and then renaming it into existence under the name you care about without the name you care about ever having a close-from-write event. That would fire off your script when a filename you were watching was unlinked and re-created instead of edited in-place, and then it's up to your script to handle the case where it's a "new" file that is really just replacing an old file. I don't even see why that should be any kind of special problem. Are some tools doing something like maintaining ooen file handles or something for every watched file??? That would break by unlink-create, but that would be insane. Perhaps these wrappers that try to simplify things are as usual just a way to break the thing they are trying to simplify. Just misguided crap. If the feature is baked into an ide or media server and failing to handle that case, well they don't have to fail to handle that case. The subsystem totally supplies the functionality. For something random adhoc where you're writing your own incrontab and handler script, it's no problem at all.
VyseofArcadia 4y ago
I've seen about a dozen versions of this now. So my question is, who watches the watchmen?
- koolba 4y ago
  If it’s running in a shell, it’d be the terminal session’s shell. If it’s running as a daemon, in a traditional Unix environment, it’d be the init process (pid 0).
  - Brian_K_White 4y ago
    Inotify is a kernel feature. Neither init or a shell or any other userspace code. The userspace code like in inotifywait or incron just tells the kernel what it wants the kernel to watch, and then exits, gone, no further parent, not even init. Ok now that I mentioned incron..., incron being a daemon is managed by init, but incron is not watching any files, not even it's own config files. incron just reads it's incrontab files and registers the desired events with the kernel, and then performs the desired actions when the kernel triggers it. It notices when it's own config files change the same way as any other inotify job, it asks the kernel to tell it when they change. incron is like cron where you write a crontab-like file that specifies a command of your choosing. Pretty much the same as this thing minus some colorful display. I guess the point of this thing would be if it provides a consistent wrapper interface over the various fs watching mechanisms on different OS's. But for linux, I've already been using incron to do this job efficiently for ages.
- cjcampbell 4y ago
  Came here for this. Thank you.
benjaminjosephw 4y ago
This project wraps `inotifywait`[0] which I didn't know about until just now. Really useful utility, thanks for sharing! [0] - https://www.man7.org/linux/man-pages/man1/inotifywait.1.html https://www.man7.org/linux/man-pages/man1/inotifywait.1.html
bin_bash 4y ago
this project hasn't had a commit in 8 years and only has a handful of stars. It confusingly shares the name with a much more popular tool from Meta—though the use case is slightly different. There are other much more popular tools that do the same thing, like entr, watchexec, and nodemon. Why is this posted?
- deleted 4y ago
  [deleted]
- Chris2048 4y ago
  Funnily enough, the real value of this post is the thread detailing all the alternatives...
- TheRealPomax 4y ago
  Small correction: watchman is a tool made and released by Facebook, not made and released by Meta. (yes, one owns the other, but Facebook employees writing and releasing code for Facebook are on Facebook's payroll, not Meta's payroll)
  - bin_bash 4y ago
    No, there is no such thing as “Facebook’s Payroll”. It’s all Meta. I’m a former Meta employee. Watchman is used heavily across the family of apps. It’s not like Alphabet that’s a holding company for different smaller companies. Watchman not yet being branded with “Meta” is a case of laziness. It doesn’t indicate it’s limited to the Blue app.
- crodjer 4y ago
  In my defense, I didn't know about Facebook's watchman then (did it even exist?). `watchman` sounded like an appropriate name to me for the task this script was for. I had not intention about gaining from folks accidentally landing on my project. Apologies if people felt mislead. I have no idea why this was suddenly posted on Hacker News yesterday.
akras14 4y ago
Edit: it may have been a different watchman, but same point still applies. It’s been a few years, so perhaps it got addressed, but this library used to introduce a “memory leak” by design, by infinity storing pointers to files being watched. It was a pain to isolate (https://techtldr.com/simple-guide-to-finding-a-javascript-memory-leak-in-node-js/ https://techtldr.com/simple-guide-to-finding-a-javascript-me...) Ultimately I ended up fixing it by writing my own version using native system calls available on Node. The moral of the story, this library is great if you are building cross platform dev tool. It is not (at least wasn’t at the time) great for long lived processes that target only one type of operating system.
vinay_ys 4y ago
Its a simple shell script wrapper on top of inotify-tools. I suppose its biggest contribution is a more appealing name - watchman.
wmichelin 4y ago
I have been using reflex https://github.com/cespare/reflex https://github.com/cespare/reflex for the same thing in some of our docker projects
- athorax 4y ago
  +1 for reflex, love being able to have a config for development Example .reflex.conf to run one command when an openapi spec changes and another when any go files change: -g "spec.yaml" -- bash -c 'make' -sr "\.go$" -- go run cmd/app/main.go Then run it with: reflex -d fancy -c .reflex.conf
dec0dedab0de 4y ago
I'm so used to making snap judgements on complex projects, and seeing that every file was last committed 8 years ago is usually a giant red flag to me. However, this is a single bash script, which definitely could be stable and forgotten about. Something like this would be more appealing as a posix shell script, but if you're using bash anyway and not trying to distribute, then stable and simple goes a long way.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
sproketboy 4y ago
Brian_K_White 4y ago
incron
tartuffe78 4y ago
I'm gonna try to use this to revert the .xcscheme file changes that Xcode makes when I switch between various Xcode versions on my machine. These happen in 10+ submodules everyday so this is great!
pyuser583 4y ago
Very cool
Zamicol 4y ago
Here was our take on the problem written in Go. https://github.com/Cyphrme/watch https://github.com/Cyphrme/watch { "path_to_watch":"example_command_to_execute_on_change.sh", }
e-dant 4y ago
Those considering doing something like this (such as the author) might consider using my library: https://github.com/e-dant/watcher https://github.com/e-dant/watcher It hooks into system APIs where viable. (Otherwise, it uses std::filesystem.) It’s meant to be as or more: - Easy to use - Efficient Than all similar libraries.
lloeki 4y ago
I've been using fswatch to great effect. The big advantage is that it's truly cross-platform, falling back to polling if no fs notification system is available as long as stat(2) works. Very composable as you just pipe its output to whatever you want (typically a while read do end), so exclusion is a awk/sed/perl/grep/rg away, and command can be as simple or complex as it needs to while the tool itself stays lean. Cool features include -o to just be notified that something changed (e.g to fire up rsync), and ability to batch changes. Also the command is a front end to libfswatch, so one can use that directly if shelling is to be foregone. https://github.com/emcrisostomo/fswatch https://github.com/emcrisostomo/fswatch
- chmaynard 4y ago
  > If you are installing fswatch using a package manager and you would like the PDF manual to be bundled into the package, please send a feature request to the package maintainer. I can't find the PDF manual in the repo. Do I need to contact the author and say please?
  - chmaynard 4y ago
    Found it: https://emcrisostomo.github.io/fswatch/doc/1.17.1/fswatch.pdf https://emcrisostomo.github.io/fswatch/doc/1.17.1/fswatch.pd...
benniomars 4y ago
Nice. I've been using yardman that's written for NodeJS and Crow written in Go. Nice to have something that's just bash.
st0le 4y ago
Is there a similar tool that can trigger a command when the output of command changes or the stdout has some kind of string? I understand this is shell script territory, but a cross-platform tool would be ideal.
m00dy 4y ago
I wrote something very similar. https://github.com/m00dy/fwe https://github.com/m00dy/fwe
pletnes 4y ago
If you like python or work on mac/win/linux, you might like https://github.com/samuelcolvin/watchfiles https://github.com/samuelcolvin/watchfiles Uses rust for performance, and is usable from the terminal or from python.
david2ndaccount 4y ago
I wrote a similar tool for macOS, using libdispatch. https://github.com/drpriver/macwatch https://github.com/drpriver/macwatch
EvanAnderson 4y ago
Does anybody have a good "execute a command when nothing changes" tool? I have some situations where I'd like to take action if, say, a file hasn't had its mtime updated in 24 hours.
- pitaj 4y ago
  Does it need to be precise? I'd probably just use an hourly cron job.
- yabones 4y ago
  Not sure if there's a way to do it instantly, but one way is to just run a find command in a cron job every X hours: find /foo/bar -type f -mtime +1 | xargs foobar
  - deadfece 4y ago
    find's own exec option is usually good enough for most things: -exec foobar {} +
thomasfl 4y ago
Shameless plug for my own take on file watchers. The filewatcher command line utility is available as a ruby gem. It exports shell variables, so you can print name of files that changes this way: $ filewatcher *.js 'echo $FILENAME' It recently reached 1 million downloads. The current maintainer, Alex Popov, is a very skilled developer working from Moscow.
jimpudar 4y ago
I haven't seen anybody here mention that you can do this with systemd path units [0],[1]. I know not everybody uses Linux (or loves systemd as much as I do) but it's a great solution if you already use systemd. [0] https://www.redhat.com/sysadmin/introduction-path-units https://www.redhat.com/sysadmin/introduction-path-units [1] https://www.freedesktop.org/software/systemd/man/systemd.path.html https://www.freedesktop.org/software/systemd/man/systemd.pat...
dabears 4y ago
Inotify can drop events when there is a burst of file changes. You can bump up the max number of queued events though. For example, if you're watching a git repo and switch to an old branch that has 1500+ changed files then inotify will likely miss some with a default configuration.
throwaway2037 4y ago
This post/share nerd-sniped me so bad. I went down a real rabbit hole reading about Linux inotify. The man pages are so well written. Example: https://man7.org/linux/man-pages/man7/inotify.7.html https://man7.org/linux/man-pages/man7/inotify.7.html I also found this Cygwin mailing list post by Erik Blake: https://cygwin.com/pipermail/cygwin/2018-May/237069.html https://cygwin.com/pipermail/cygwin/2018-May/237069.html And patches to make Cygwin support inotify_init() and friends implemented on top of the Windows native API are certainly welcome. I think I just found my next personal project...
Maursault 4y ago
> I have always wanted a watch tool which instead of being based on time (the watch command) will be based on space (files). Can't this functionality be duplicated with a while loop and du? while $(du -s file.log) < 200; do sleep 0; done; [some command here]
timvisee 4y ago
https://github.com/timvisee/dotfiles/blob/6f5af040632586929d114439280a6577fcc88ce6/bash/bash_functions#L117-L124 https://github.com/timvisee/dotfiles/blob/6f5af040632586929d...
crodjer 4y ago
Hello everyone, I am the author of this project. Not sure what generated interest in this. This was a project I did for my personal use case. But I haven't been using it since years. I'd recommend [entr](https://github.com/clibs/entr https://github.com/clibs/entr) for the use case watchman was to serve.
prirai 4y ago
Btw I use entr. Cool little lightweight tool to do just that.

248 points by ColinWright 4y ago

Watchman works great, but the CLI UX is... really kinda dreadful in my opinion. Trying to get it usable in diverse scenarios kinda reveals that the author has a very particular way it's meant to be used and it seems needlessly frustrating to use it otherwise.

BrainVirus 4y ago

This command seems like a no-brainer that should be in all OSes by default. It's mind-boggling that most systems don't have a standard tool for observing file changes. The only language I know of that has a platform-agnostic API for this is Factor.

dspillett 4y ago
The required kernel hooks exist in pretty much any common OS these days, it is a user-space tool that is sometimes missing. It may not be installed by default, but inotifywait is available in common Linux distributions, usually in a package called something like ionotify-tools, and has been for over a decade-ana-half IIRC. It'll work under WSL on Windows too, though only for ext4 devices not bits of the Windows filesystem made available to Linux. I can't speak to what other OSs include by default, but as every major OS has a different API for how to register a listener & how it gets messages no built-in tool is going to be cross-platform. There are third party tools which present more cross-platform consistency, most notably https://github.com/emcrisostomo/fswatch#readme https://github.com/emcrisostomo/fswatch#readme (also available in common Linux distros, just an apt install away in Debian for instance).

jitl 4y ago

Not to be confused with Facebook’s file watch daemon, also names watchman, which does the same sort of thing but is more complicated. There’s a bunch of tools that integrate Facebook’s watchman for more efficient change tracking. Advantages of Facebook’s watchman: 1. Implements efficient file system event watches on macOS and Windows 2. IPC/daemon system reduces resource use because overlapping watches/triggers don’t use more inotify slots. 3. Denounces / waits for changes to settle 4. Client libraries in a few different languages for scripting purposes https://facebook.github.io/watchman/ https://facebook.github.io/watchman/

HelloNurse 4y ago
5. Windows is supported.
deleted 4y ago
[deleted]
dec0dedab0de 4y ago
Also not to be confused with watchdog and it's command watchmedo, which is what I thought this was when I first read the headline. I don't know what the differences are, but I use watchdog to restart my local celery workers when I make changes. It's scripted out so I don't ever use the command directly. https://pypi.org/project/watchdog/ https://pypi.org/project/watchdog/ https://github.com/gorakhargosh/watchdog https://github.com/gorakhargosh/watchdog EDIT: Now I'm wondering how often I miss out on something because I confuse it for something I already use. Maybe there is room for a "things I use" or "things I know about" browser plugin.
- ghassanmas 4y ago
  Also to note "watchdog" the term. Is used also in hardware/software context. Usually as a way to mitgate or recover system faulate. I first I heard of this term while taking in a course about PIC microcontroller[1]. > The Watchdog Timer in the PIC16F819 The watchdog timer can be a real source of pain and can also make a PIC system incredibly robust and reliable. But what exactly is the watchdog timer? Simply put, the watchdog timer is a hardware timer in the PIC that, if not constantly reset by the software, will cause the PIC to reset. This feature can be incredibly useful if the PIC should hang due to a hardware or software issue and guarantees that the PIC will restart from the beginning. Not only does it reset the system, it also flags a bit that can be used to determine if the system just crashed. Ref: https://maker.pro/pic/tutorial/how-to-get-started-with-pic-microcontrollers-timer0-and-the-watchdog-timer https://maker.pro/pic/tutorial/how-to-get-started-with-pic-m... [1]: PIC microcontroller, is a small hardware simliar to arduino but more on a lower level.
- damagednoob 4y ago
  Also not to be confused with watchexec (my tool of choice for executing commands on file changes): https://github.com/watchexec/watchexec https://github.com/watchexec/watchexec

GRBurst 4y ago

For those use-cases I use entr (https://github.com/clibs/entr https://github.com/clibs/entr). In what sense does watchman differ to it? Didn't see anything related to this at first glance :-)

deleted 4y ago
[deleted]
everybodyknows 4y ago
The page linked above says: > WARNING: This is a (possibly outdated and/or unmaintained) fork of https://github.com/eradman/entr https://github.com/eradman/entr .
- deleted 4y ago
  [deleted]
DotaFan 4y ago
OP did say he did this for learning purpose.

basemi 4y ago

I've used `incron`[0] for that purpose, with success. Packaged for most distros I think [0]https://github.com/ar-/incron https://github.com/ar-/incron

IncRnd 4y ago

What does this add to inotifywait? That does exactly the same as this project, which uses inotifywait to reimplement inotifywait. For example: dnf install inotify-tools or apt-get install inotify-tools

masukomi 4y ago
ifnotifywait doesn't exist on macOS, so in addition to that not being an option on macos, this would be something you could write tools that a team who used multiple platforms, or a person with setups on multiple os's, could use. I suspect this might even be usable on windows. [edit: fswatch looks very promising, and may be a better choice than either ifnotifywait or watchman]
- IncRnd 4y ago
  You are confused about how the tool works. This project uses inotifywait from inotify tools to reimplement what inotifywait already does. The linked article page at github clearly says this. Adding a new layer atop inotify tools doesn't make that package automatically run on new platforms. There are already tools on macos that use the native FSEvents API for watching for file changes and executing scripts.
- awinder 4y ago
  This wraps inotifywait, so it doesn’t really address cross-platform needs. The unique features in this over straight inotifywait is plugging together the command you want to execute, and some event dedupe logic. Grand majority of the code is around colorized logs, command docs, and some options handling code which mostly just gets piped to inotifywait.
- wwweston 4y ago
  Discovered fswatch recently, it's proved very useful, my favorite use being replacing a few situations where I was using sshfs with fswatch+rsync: no network lag on save/load, can even work offline, but everything still makes its way to the network destination!
  - 4y ago

majodev 4y ago

Very happy user of https://github.com/watchexec/watchexec https://github.com/watchexec/watchexec (rust), can recommend.

loevborg 4y ago
Yes, this is the one I reach for always. It works reliably AFAICT, runs on all platforms, is simple enough and still has the necessary features.
maximilianroos 4y ago
Very much agree. And the maintainer has been super responsive.
crodjer 4y ago
This looks great! I'll also check this out as someone who pivoted to rust tooling in the recent years.

amelius 4y ago

From my experience, these types of notification handlers are very fragile and tend to fail every so often. Do not use if your life depends on it.

remram 4y ago
From my experience, they break every time if you are making changes with tools that replace the file rather than change it in-place, like most text editors.
- amelius 4y ago
  In that case, you might want to monitor the directory too, not just the file. I believe this is even how it is supposed to work.
  - remram 4y ago
    Yeah, and then filter out the events in the directory to find out when the correct file changed, etc. That is exactly what I would want the tool to abstract for me.
- Brian_K_White 4y ago
  At least with inotify, ie, incron, this is just a matter of setting a less stupid watch. You can ask the kernel to watch for several different specific kinds of fs events, either at the file or directory level. Detecting when a file is either created, or closed from writing, is no problem at all. Ie, handles the case of writing a file normally, editing a file, and renaming a file into exitence without having opened it for writing (that happened while it had some other name you weren't watching). This would handle unlinking and recreating the same as editing or uploading. You can watch specifically for the close event only, and even more specifically for close-from-write-mode to detect updates but ignore until the update is done. And at the same time you can also watch for creation, which handles creating a file under some other name and then renaming it into existence under the name you care about without the name you care about ever having a close-from-write event. That would fire off your script when a filename you were watching was unlinked and re-created instead of edited in-place, and then it's up to your script to handle the case where it's a "new" file that is really just replacing an old file. I don't even see why that should be any kind of special problem. Are some tools doing something like maintaining ooen file handles or something for every watched file??? That would break by unlink-create, but that would be insane. Perhaps these wrappers that try to simplify things are as usual just a way to break the thing they are trying to simplify. Just misguided crap. If the feature is baked into an ide or media server and failing to handle that case, well they don't have to fail to handle that case. The subsystem totally supplies the functionality. For something random adhoc where you're writing your own incrontab and handler script, it's no problem at all.

VyseofArcadia 4y ago

I've seen about a dozen versions of this now. So my question is, who watches the watchmen?

koolba 4y ago
If it’s running in a shell, it’d be the terminal session’s shell. If it’s running as a daemon, in a traditional Unix environment, it’d be the init process (pid 0).
- Brian_K_White 4y ago
  Inotify is a kernel feature. Neither init or a shell or any other userspace code. The userspace code like in inotifywait or incron just tells the kernel what it wants the kernel to watch, and then exits, gone, no further parent, not even init. Ok now that I mentioned incron..., incron being a daemon is managed by init, but incron is not watching any files, not even it's own config files. incron just reads it's incrontab files and registers the desired events with the kernel, and then performs the desired actions when the kernel triggers it. It notices when it's own config files change the same way as any other inotify job, it asks the kernel to tell it when they change. incron is like cron where you write a crontab-like file that specifies a command of your choosing. Pretty much the same as this thing minus some colorful display. I guess the point of this thing would be if it provides a consistent wrapper interface over the various fs watching mechanisms on different OS's. But for linux, I've already been using incron to do this job efficiently for ages.
cjcampbell 4y ago
Came here for this. Thank you.

benjaminjosephw 4y ago

This project wraps `inotifywait`[0] which I didn't know about until just now. Really useful utility, thanks for sharing! [0] - https://www.man7.org/linux/man-pages/man1/inotifywait.1.html https://www.man7.org/linux/man-pages/man1/inotifywait.1.html

bin_bash 4y ago

this project hasn't had a commit in 8 years and only has a handful of stars. It confusingly shares the name with a much more popular tool from Meta—though the use case is slightly different. There are other much more popular tools that do the same thing, like entr, watchexec, and nodemon. Why is this posted?

deleted 4y ago
[deleted]
Chris2048 4y ago
Funnily enough, the real value of this post is the thread detailing all the alternatives...
TheRealPomax 4y ago
Small correction: watchman is a tool made and released by Facebook, not made and released by Meta. (yes, one owns the other, but Facebook employees writing and releasing code for Facebook are on Facebook's payroll, not Meta's payroll)
- bin_bash 4y ago
  No, there is no such thing as “Facebook’s Payroll”. It’s all Meta. I’m a former Meta employee. Watchman is used heavily across the family of apps. It’s not like Alphabet that’s a holding company for different smaller companies. Watchman not yet being branded with “Meta” is a case of laziness. It doesn’t indicate it’s limited to the Blue app.
crodjer 4y ago
In my defense, I didn't know about Facebook's watchman then (did it even exist?). `watchman` sounded like an appropriate name to me for the task this script was for. I had not intention about gaining from folks accidentally landing on my project. Apologies if people felt mislead. I have no idea why this was suddenly posted on Hacker News yesterday.

akras14 4y ago

Edit: it may have been a different watchman, but same point still applies. It’s been a few years, so perhaps it got addressed, but this library used to introduce a “memory leak” by design, by infinity storing pointers to files being watched. It was a pain to isolate (https://techtldr.com/simple-guide-to-finding-a-javascript-memory-leak-in-node-js/ https://techtldr.com/simple-guide-to-finding-a-javascript-me...) Ultimately I ended up fixing it by writing my own version using native system calls available on Node. The moral of the story, this library is great if you are building cross platform dev tool. It is not (at least wasn’t at the time) great for long lived processes that target only one type of operating system.

vinay_ys 4y ago

Its a simple shell script wrapper on top of inotify-tools. I suppose its biggest contribution is a more appealing name - watchman.

wmichelin 4y ago

I have been using reflex https://github.com/cespare/reflex https://github.com/cespare/reflex for the same thing in some of our docker projects

athorax 4y ago
+1 for reflex, love being able to have a config for development Example .reflex.conf to run one command when an openapi spec changes and another when any go files change: -g "spec.yaml" -- bash -c 'make' -sr "\.go$" -- go run cmd/app/main.go Then run it with: reflex -d fancy -c .reflex.conf

dec0dedab0de 4y ago

I'm so used to making snap judgements on complex projects, and seeing that every file was last committed 8 years ago is usually a giant red flag to me. However, this is a single bash script, which definitely could be stable and forgotten about. Something like this would be more appealing as a posix shell script, but if you're using bash anyway and not trying to distribute, then stable and simple goes a long way.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

sproketboy 4y ago

Brian_K_White 4y ago

incron

tartuffe78 4y ago

I'm gonna try to use this to revert the .xcscheme file changes that Xcode makes when I switch between various Xcode versions on my machine. These happen in 10+ submodules everyday so this is great!

pyuser583 4y ago

Very cool

Zamicol 4y ago

Here was our take on the problem written in Go. https://github.com/Cyphrme/watch https://github.com/Cyphrme/watch { "path_to_watch":"example_command_to_execute_on_change.sh", }

e-dant 4y ago

Those considering doing something like this (such as the author) might consider using my library: https://github.com/e-dant/watcher https://github.com/e-dant/watcher It hooks into system APIs where viable. (Otherwise, it uses std::filesystem.) It’s meant to be as or more: - Easy to use - Efficient Than all similar libraries.

lloeki 4y ago

I've been using fswatch to great effect. The big advantage is that it's truly cross-platform, falling back to polling if no fs notification system is available as long as stat(2) works. Very composable as you just pipe its output to whatever you want (typically a while read do end), so exclusion is a awk/sed/perl/grep/rg away, and command can be as simple or complex as it needs to while the tool itself stays lean. Cool features include -o to just be notified that something changed (e.g to fire up rsync), and ability to batch changes. Also the command is a front end to libfswatch, so one can use that directly if shelling is to be foregone. https://github.com/emcrisostomo/fswatch https://github.com/emcrisostomo/fswatch

chmaynard 4y ago
> If you are installing fswatch using a package manager and you would like the PDF manual to be bundled into the package, please send a feature request to the package maintainer. I can't find the PDF manual in the repo. Do I need to contact the author and say please?
- chmaynard 4y ago
  Found it: https://emcrisostomo.github.io/fswatch/doc/1.17.1/fswatch.pdf https://emcrisostomo.github.io/fswatch/doc/1.17.1/fswatch.pd...

benniomars 4y ago

Nice. I've been using yardman that's written for NodeJS and Crow written in Go. Nice to have something that's just bash.

st0le 4y ago

Is there a similar tool that can trigger a command when the output of command changes or the stdout has some kind of string? I understand this is shell script territory, but a cross-platform tool would be ideal.

m00dy 4y ago

I wrote something very similar. https://github.com/m00dy/fwe https://github.com/m00dy/fwe

pletnes 4y ago

If you like python or work on mac/win/linux, you might like https://github.com/samuelcolvin/watchfiles https://github.com/samuelcolvin/watchfiles Uses rust for performance, and is usable from the terminal or from python.

david2ndaccount 4y ago

I wrote a similar tool for macOS, using libdispatch. https://github.com/drpriver/macwatch https://github.com/drpriver/macwatch

EvanAnderson 4y ago

Does anybody have a good "execute a command when nothing changes" tool? I have some situations where I'd like to take action if, say, a file hasn't had its mtime updated in 24 hours.

pitaj 4y ago
Does it need to be precise? I'd probably just use an hourly cron job.
yabones 4y ago
Not sure if there's a way to do it instantly, but one way is to just run a find command in a cron job every X hours: find /foo/bar -type f -mtime +1 | xargs foobar
- deadfece 4y ago
  find's own exec option is usually good enough for most things: -exec foobar {} +

thomasfl 4y ago

Shameless plug for my own take on file watchers. The filewatcher command line utility is available as a ruby gem. It exports shell variables, so you can print name of files that changes this way: $ filewatcher *.js 'echo $FILENAME' It recently reached 1 million downloads. The current maintainer, Alex Popov, is a very skilled developer working from Moscow.

jimpudar 4y ago

I haven't seen anybody here mention that you can do this with systemd path units [0],[1]. I know not everybody uses Linux (or loves systemd as much as I do) but it's a great solution if you already use systemd. [0] https://www.redhat.com/sysadmin/introduction-path-units https://www.redhat.com/sysadmin/introduction-path-units [1] https://www.freedesktop.org/software/systemd/man/systemd.path.html https://www.freedesktop.org/software/systemd/man/systemd.pat...

dabears 4y ago

Inotify can drop events when there is a burst of file changes. You can bump up the max number of queued events though. For example, if you're watching a git repo and switch to an old branch that has 1500+ changed files then inotify will likely miss some with a default configuration.

throwaway2037 4y ago

This post/share nerd-sniped me so bad. I went down a real rabbit hole reading about Linux inotify. The man pages are so well written. Example: https://man7.org/linux/man-pages/man7/inotify.7.html https://man7.org/linux/man-pages/man7/inotify.7.html I also found this Cygwin mailing list post by Erik Blake: https://cygwin.com/pipermail/cygwin/2018-May/237069.html https://cygwin.com/pipermail/cygwin/2018-May/237069.html And patches to make Cygwin support inotify_init() and friends implemented on top of the Windows native API are certainly welcome. I think I just found my next personal project...

Maursault 4y ago

> I have always wanted a watch tool which instead of being based on time (the watch command) will be based on space (files). Can't this functionality be duplicated with a while loop and du? while $(du -s file.log) < 200; do sleep 0; done; [some command here]

timvisee 4y ago

https://github.com/timvisee/dotfiles/blob/6f5af040632586929d114439280a6577fcc88ce6/bash/bash_functions#L117-L124 https://github.com/timvisee/dotfiles/blob/6f5af040632586929d...

crodjer 4y ago

Hello everyone, I am the author of this project. Not sure what generated interest in this. This was a project I did for my personal use case. But I haven't been using it since years. I'd recommend [entr](https://github.com/clibs/entr https://github.com/clibs/entr) for the use case watchman was to serve.

prirai 4y ago

Btw I use entr. Cool little lightweight tool to do just that.