8 ms·
Show HN: Filesystem Watcher
An arbitrary filesystem event watcher which is:
- simple
- efficient
- dependency free
- runnable anywhere with a filesystem
- header only
Watcher is extremely efficient. In most cases, even when scanning millions of paths, this library uses a near-zero amount of resources.
Watcher is simple. The library exposes a single function and a single object. That is all.
Happy hacking.
- deleted 4y ago[deleted]
- e-dant 4y ago
- e-dant 4y ago
- widdershins 4y agoLooks great! I'm wondering what operating systems are supported. I'm assuming Linux. What about macOS and Windows?
- dljsjr 4y agohttps://github.com/e-dant/watcher/blob/989147b183ee0547d71a14681c0b72d5b12a30e0/include/watcher/platform.hpp#L8 https://github.com/e-dant/watcher/blob/989147b183ee0547d71a1... Looks like it works on quite a few systems including Android and iOS.
- e-dant 4y agoAll are supported. Although, to be more efficient, I need to write system API calls for Windows. That will be the 1.0 release.
- hackyhacky 4y ago[flagged]
- sk0g 4y agoThey were responding to a comment asking about Linux, MacOS, and Windows support. Needless pedantry is one thing, but deliberately misconstruing a discussion to support said pedantry is sad. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- OnlyMortal 4y agoIt uses FSEvents on the Mac. It can do dumb polling too.
- GordonS 4y agoHow does this work under the covers on Linux? Is it using eBPF, or is it simply an abstraction over inotify? I'm particularly interested in something like this, but which will include information about what process made the change, and which user it was running as at the time.
- bertman 4y ago>or is it simply an abstraction over inotify? Looks like it: https://github.com/e-dant/watcher/blob/989147b183ee0547d71a14681c0b72d5b12a30e0/include/watcher/adapter/linux/watch.hpp#L9 https://github.com/e-dant/watcher/blob/989147b183ee0547d71a1...
- e-dant 4y agoI’ve gone back and forth with inotify on Linux. Ned14 gave a great rundown of the ideal next steps for a best-possible implementation. You can check out issue/10 for a full description of how it works now, why neither inotify nor our current solution is ideal, and where the project will be going next.
- GordonS 4y agoI'm sorry, I don't know what issue/10 means? Is it an e-zine or something? (apologies if this is obvious, I'm extremely tired!)
- mikulas_florek 4y agoLooking at Win32, it scans the whole directory periodically, right? I must miss something, but how can that be called efficient?
- rasz 4y agoWait, it doesnt hook the OS file handling routines? it actually manually rescans the filesystem?
- deleted 4y ago[deleted]
- e-dant 4y agoIt does one or the other. There are concerns about OS filesystem event hooks. The current solution isn’t ideal, and is being addressed here: https://github.com/e-dant/watcher/issues/10 https://github.com/e-dant/watcher/issues/10
- mikulas_florek 4y agonot on windows (only platform I checked/care about)
- dwringer 4y agoSee here: https://news.ycombinator.com/item?id=33247735 https://news.ycombinator.com/item?id=33247735
- e-dant 4y agoIt’s efficient because it beats kqueue while reporting events accurately. A proper benchmarking program is in the works, however manual testing does show only minimal resource usage. For more, see this issue: https://github.com/e-dant/watcher/issues/10 https://github.com/e-dant/watcher/issues/10
- mikulas_florek 4y ago
- remram 4y agoDoes it keep working when files get overwritten by moving another over it, like some Linux text editors do?
- e-dant 4y agoI should test this. I haven’t seen a problem with that so far in my personal usage, so I’m inclined to say probably. That’s a good test case. I’ll make an issue.
- sgerenser 4y agoThis was a case that tripped me up when using Qt's QFileSystemWatcher. Many apps implement "atomic updates" to files by writing out to a new file (e.g. foo.txt.temp), then moving that file on top of the old one. As far as QFileSystemWatcher was concerned, the old file was deleted and its job is done (just because the new one has the same name doesn't mean it's the same file). So you have to watch the parent directory and manually implement checking for a specific file by name when the parent directory's contents changed.
- deleted 4y ago[deleted]
- rasz 4y agoLooks like it does exactly what you can get out of Everything (https://www.voidtools.com https://www.voidtools.com) Index Journal https://www.voidtools.com/forum/viewtopic.php?t=9792 https://www.voidtools.com/forum/viewtopic.php?t=9792 but programmatically and its highly scriptable, pretty cool. Will definitely add it to my arsenal of troubleshooting tools. Edit: never mind, This tool is manually scanning the filesystem instead of listening to OS events https://github.com/e-dant/watcher/blob/989147b183ee0547d71a14681c0b72d5b12a30e0/include/watcher/adapter/windows/watch.hpp https://github.com/e-dant/watcher/blob/989147b183ee0547d71a1...
- mellosouls 4y agoVoidtools gives security warnings in my browser, fwiw.
- rasz 4y agoOf course it does, its competing with Microsoft by providing actually working instant local search.
- mellosouls 4y agoMy chrome browser.
- rasz 4y ago> instant local search still checks out :) But I checked just to be sure and no warning in Version 106.0.5249.91 (Official Build) (32-bit). Maybe its your corporate baby content web filter?
- mellosouls 4y agoNo, its a certificate issue; probably minor, but a server-side thing to attend to by the looks of it.
- awestroke 4y agoSee also: sane - for node watchexec - rust based, static binary
- christophilus 4y agoReflex for go: https://github.com/cespare/reflex https://github.com/cespare/reflex
- kevincox 4y agoFor CLI usage I found that the best option was instead of watching my source directory just to watch a magic file. Then I configured my editor to touch that file when saving. This has a few benefits: 1. No need to worry about which files to watch or ignoring build outputs. 2. Works with every project with no setup. 3. Easy to trigger a re-run without actually changing a file. 4. Always runs after all files are saved instead of starting after the first file is saved and racing the rest. 5. Infinitely scalable.
- matijs 4y agoOut of curiosity, what editor do you use and how do you make sure 4. happens when for example using ‘save all’?
- kevincox 4y agoI'm currently using neovim so it is pretty trivial to add save hooks. Although the approach I am currently using is just a custom shortcut that saves all files and touches the file. This way only explicit saves by me trigger the rerun. My current setup is documented here but it's easy to tweak to your prefered workflow. https://kevincox.ca/2022/06/14/small-tools/#w https://kevincox.ca/2022/06/14/small-tools/#w
- cmovq 4y agoAny reason for making delay_ms a template parameter? A compiler should be able to optimize passing a constant as a regular function argument. And if it’s not optimized I assume a variable delay wouldn’t affect much?
- e-dant 4y agoNo perfectly good reason. I will look into that before version 1.
- queuebert 4y agoIn the early days of Linux, there was a tool that saved file info to a floppy. Then you would write protect the floppy and leave it in a drive, and the tool would periodically compare OS files with that to detect alteration. I can't for the life of me remember the name, though. It was great for hardened systems.
- stonogo 4y agoTripwire. https://www.linuxjournal.com/article/8758 https://www.linuxjournal.com/article/8758
- waynesonfire 4y agoforgot "written in C++" .. unless you're trying to get views, better to leave that part out.
- diffxx 4y agoI have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into the underlying kernel events like kqueue, inotify, fsevents, ReadDirectoryChangesW, etc and report events Option 1 uses a lot of CPU and memory (the map storing the paths being monitored could easily grow to be tens or even hundreds of megabytes if many files are being monitored, which is often the case in large source projects). I have seen tools that use polling with a 100ms interval continuously burn 50% of cpu monitoring a modest sized directory with tens of thousands of files. Option 2 theoretically would use less memory and little to no cpu, but in practice, the story is more complicated. If you are using an inotify or kqueue like api, you will have to store handles for all of the paths that are being monitored, which can take a significant amount of memory. On macos, the file system events are not accurate in the sense that you can't trust the type of event. It doesn't reliably distinguish between creation and modification events. So if you want to know specifically what kind of event happened, you end up back in case 1 where you have to store an in memory representation of the file system and diff against the in memory representation and the current file system state when you detect an event. For some use cases, you may not care to distinguish between creations and modifications and can get away with a lower memory, but less accurate, solution. In my experience, getting all of this right is much more difficult than it appears at first glance. Good luck to you.
- e-dant 4y agoIt’s difficult to get it perfectly right. There is ongoing work attempting to make it more perfect. I expect a year or two before this is complete. For now though, it does do what it says. The tests I’ve run show that it is accurate over large amounts of events and time. For under 1 million files and/or directories, it uses a near-zero amount of resources. Testing on older processors shows similarly positive results. But this is so far from perfect. This is only the groundwork. Most of the bugs have yet to be discovered. The platform support, more often than not, uses the safe “baseline” watcher in favor of accuracy. Ned14 of Boost fame has given the project some expert advice which will help it along smoothly.
- dyerjohn 4y ago"Watcher is extremely efficient. In most cases, even when scanning millions of paths, this library uses a near-zero amount of resources." Yea, maybe or maybe not and my first guess is maybe not. This needs at least some bullet points on HOW it does this so efficiently so that I'll keep looking. A blanket statement like this means "they hope it is efficient" or "They want it to be efficient" or "It's good in some scenarios but not others". With those additional bits, I have a reason to dig around the source.
- e-dant 4y agoYou are right. I’ll make sure to give a deeper breakdown in the readme.
- chasil 4y agoIs this just using inotify on Linux? If so, there are equivalent options, including systemd path units, incron, and the inotifywait utility, in addition to the C API. The "man systemd.path" page does list explicit limitations of this kernel system call: "Internally, path units use the inotify(7) API to monitor file systems. Due to that, it suffers by the same limitations as inotify, and for example cannot be used to monitor files or directories changed by other machines on remote NFS file systems." (Files modified by mmap() also don't trigger events.) https://www.linuxjournal.com/content/linux-filesystem-events-inotify https://www.linuxjournal.com/content/linux-filesystem-events... Windows busybox also has an inotifyd, which appears to do something similar.
- roeles 4y agoShould this work over NFS or SMB?
- bigmattystyles 4y agoWhen I last tried to implement this, by far the toughest part was making sure the file that’s been newly detected is done being written to. On ntfs I couldn’t find a good technique, even last modified time was not reliable. I had to watch it for changes myself.
- nomel 4y agoIs "last modified" the time of the beginning of the write?
- bigmattystyles 4y agoNever even thought of that; I don’t know. I assumed it was when a write was done. Whatever that means, I don’t know either.
- infogulch 4y agoI've done this by watching the NTFS journal which is surprisingly efficient. First I scanned the whole journal for filesystem metadata and dumped it into a SQLite database (which took about a minute), then kept it up to date which took virtually no resources. This was an absurdly faster way to search by file name, a search across the whole FS came back in milliseconds instead of Explorer's multiple minutes.
- jonhohle 4y agoHow does this compare to something like famd(8) and if it’s a marked improvement, could the techniques used here be backported to FAM?
- POPOSYS 4y agoIs there a well-tested, reliable, flexible and good working tool for windows that does the same and can be installed as a service? Just watch a directory and do something that I can configure easily with a textfile?
- POPOSYS 4y agoWell, I am not too lazy to search but I was interested in your experience, especially with reliability. This one looks interesting: https://github.com/emcrisostomo/fswatch https://github.com/emcrisostomo/fswatch