4 ms·

It’s difficult to get it perfectly right. There is ongoing work attempting to make it more perfect. I expect a year or two before this is complete. For now t

by e-dant 4y ago

It’s difficult to get it perfectly right. There is ongoing work attempting to make it more perfect. I expect a year or two before this is complete. For now though, it does do what it says. The tests I’ve run show that it is accurate over large amounts of events and time. For under 1 million files and/or directories, it uses a near-zero amount of resources. Testing on older processors shows similarly positive results. But this is so far from perfect. This is only the groundwork. Most of the bugs have yet to be discovered. The platform support, more often than not, uses the safe “baseline” watcher in favor of accuracy. Ned14 of Boost fame has given the project some expert advice which will help it along smoothly.

diffxx 4y ago
What do you mean near-zero? You said that inotify doesn't work (and ned14 offers his comments about it). If you are using polling, I do not understand how your approach could be using non-zero amount of resources. Let's say you are monitoring a directory with 1 million files, how can you store the state in less than 20MB of memory (which is about the most optimistic lower bound that I can think of)? What is your secret sauce? Do you mean there is no overhead beyond the baseline watcher? But what about the overhead of the baseline watcher itself? For what it's worth, in spite of ned14's comments, I have never seen inotify fail in practice (except for if it hits the os file descriptor limits in which case it does fail noisily). The tool I wrote uses inotify for linux. It is used by thousands of developers every day as part of an editor integration and there are no open issues about dropped file events. Your time frame is probably about right. It took me about a year to work through all the edge cases.
- e-dant 4y ago
  Near-zero is a bit loose. It keeps a relatively compact in-memory representation. You’re about right with your estimate. Having measured just now, it’s about 30mb for 1 million directories. The baseline Watcher’s efficiency has a wide spread. When there are many thousands of nested subdirectories, the CPU approaches the limit of the thread it’s on. Flatter directories, or many files without nested subdirectories, do not have nearly as much of an effect. I’ve seen it run on around 10 million paths on a very flat test directory. So, near-zero is somewhat misleading. There’s a wide spread in efficiency. It was my judgement that deeply nested directory trees were far less common in practice then, so I wrote “near-zero” in the optimistic case. It uses polling under the hood (at least, I’m sure it does. It uses whatever std::filesystem uses, which is almost certainly polling).

by e-dant 4y ago

diffxx 4y ago

What do you mean near-zero? You said that inotify doesn't work (and ned14 offers his comments about it). If you are using polling, I do not understand how your approach could be using non-zero amount of resources. Let's say you are monitoring a directory with 1 million files, how can you store the state in less than 20MB of memory (which is about the most optimistic lower bound that I can think of)? What is your secret sauce? Do you mean there is no overhead beyond the baseline watcher? But what about the overhead of the baseline watcher itself? For what it's worth, in spite of ned14's comments, I have never seen inotify fail in practice (except for if it hits the os file descriptor limits in which case it does fail noisily). The tool I wrote uses inotify for linux. It is used by thousands of developers every day as part of an editor integration and there are no open issues about dropped file events. Your time frame is probably about right. It took me about a year to work through all the edge cases.

e-dant 4y ago
Near-zero is a bit loose. It keeps a relatively compact in-memory representation. You’re about right with your estimate. Having measured just now, it’s about 30mb for 1 million directories. The baseline Watcher’s efficiency has a wide spread. When there are many thousands of nested subdirectories, the CPU approaches the limit of the thread it’s on. Flatter directories, or many files without nested subdirectories, do not have nearly as much of an effect. I’ve seen it run on around 10 million paths on a very flat test directory. So, near-zero is somewhat misleading. There’s a wide spread in efficiency. It was my judgement that deeply nested directory trees were far less common in practice then, so I wrote “near-zero” in the optimistic case. It uses polling under the hood (at least, I’m sure it does. It uses whatever std::filesystem uses, which is almost certainly polling).