4 ms·

Is inotify broken? What should be used instead of it to watch for filesystem events?

rwaksmunski 4y ago
I don't know man, I've given up and handed it over to someone more senior. Their port was half the performance and the project was killed. We kept on using FreeBSD for the product.
- nextaccountic 4y ago
  Is it broken regarding to correctness or just performance? I mean, does inotify have some obscure race condition or drop events or duplicate events or something
  - rwaksmunski 4y ago
    > Is it broken regarding to correctness or just performance? Yes. This blog entry explains it much better then I could: https://wingolog.org/archives/2018/05/21/correct-or-inotify-pick-one https://wingolog.org/archives/2018/05/21/correct-or-inotify-...
- Beltalowda 4y ago
  For watching filesystem events FreeBSD's kqueue is much more difficult to use correctly though, and in my experience it certainly doesn't outperform inotify. inotify (and now, fanotify) are specifically designed to watch for filesystem events, whereas kqueue is a generic event system used for many things. In many ways that's better, but it this lack of specialisation also comes with some drawbacks.
  - rwaksmunski 4y ago
    From what I can recall, the performance issues were on the network side. I suspected accept() calls not scaling right, but instrumentation on Linux was quite poor at that time. They had nothing like dtrace back then so I can't be certain.
jgerrish 4y ago
Jesus, is it sad when I hear senior now I don't dream of leading the design of big systems? I think of stakeholder meetings and fighting business requirements? I had an inotify project in process for creating a better developer experience. Seeing this hit Hacker News tells me it's a fucking political landmine. Sigh. Don't worry, I'll have the strength to argue the proper technical solution that still satisfies business needs. That's the important thing!
- nextaccountic 4y ago
  I don't understand what you mean. Anyway I used inotify multiple times in various programming languages and it was always fine, I'm a bit surprised that it's considered "broken" in any way. I mean, is it broken regarding to correctness or just performance?
  - jgerrish 4y ago
    I don't consider it broken, I was referring to political will to build the healthy base of a developer ecosystem. But, you've inspired me to do a deeper historical and technical deep dive of the filesystem event space. Thank you. Have a great weekend!
    - jgerrish 4y ago
      And I do understand. I am one who reached out. Thank you for showing me. They understand who I am. Have a good week!
mastax 4y ago
http://wingolog.org/archives/2018/05/21/correct-or-inotify-pick-one http://wingolog.org/archives/2018/05/21/correct-or-inotify-p...
- nextaccountic 4y ago
  Thanks! Does fanotify fix those issues?
  - scaramanga 4y ago
    Lose events due to inotify buffer overflow, or provide backpressure on entire vfs and allow a watcher to DoS the entire system. Pick one. Similar "minor implementation issues" occur when considering how to fix all of the other "bugs." It's fine if you don't like the design trade-offs of a particular solution. But pretending that they aren't trade-offs, or they aren't valid choices, is just going to leave people shaking their heads at you. Edit: and by "you" i mean the authors of these articles. Although I think Marek is a very talented guy and probably just a bit hyperbolic with his titles.

I don't know man, I've given up and handed it over to someone more senior. Their port was half the performance and the project was killed. We kept on using FreeBSD for the product.

nextaccountic 4y ago
Is it broken regarding to correctness or just performance? I mean, does inotify have some obscure race condition or drop events or duplicate events or something
- rwaksmunski 4y ago
  > Is it broken regarding to correctness or just performance? Yes. This blog entry explains it much better then I could: https://wingolog.org/archives/2018/05/21/correct-or-inotify-pick-one https://wingolog.org/archives/2018/05/21/correct-or-inotify-...
Beltalowda 4y ago
For watching filesystem events FreeBSD's kqueue is much more difficult to use correctly though, and in my experience it certainly doesn't outperform inotify. inotify (and now, fanotify) are specifically designed to watch for filesystem events, whereas kqueue is a generic event system used for many things. In many ways that's better, but it this lack of specialisation also comes with some drawbacks.
- rwaksmunski 4y ago
  From what I can recall, the performance issues were on the network side. I suspected accept() calls not scaling right, but instrumentation on Linux was quite poor at that time. They had nothing like dtrace back then so I can't be certain.

jgerrish 4y ago

Jesus, is it sad when I hear senior now I don't dream of leading the design of big systems? I think of stakeholder meetings and fighting business requirements? I had an inotify project in process for creating a better developer experience. Seeing this hit Hacker News tells me it's a fucking political landmine. Sigh. Don't worry, I'll have the strength to argue the proper technical solution that still satisfies business needs. That's the important thing!

nextaccountic 4y ago
I don't understand what you mean. Anyway I used inotify multiple times in various programming languages and it was always fine, I'm a bit surprised that it's considered "broken" in any way. I mean, is it broken regarding to correctness or just performance?
- jgerrish 4y ago
  I don't consider it broken, I was referring to political will to build the healthy base of a developer ecosystem. But, you've inspired me to do a deeper historical and technical deep dive of the filesystem event space. Thank you. Have a great weekend!
  - jgerrish 4y ago
    And I do understand. I am one who reached out. Thank you for showing me. They understand who I am. Have a good week!

mastax 4y ago

http://wingolog.org/archives/2018/05/21/correct-or-inotify-pick-one http://wingolog.org/archives/2018/05/21/correct-or-inotify-p...

nextaccountic 4y ago
Thanks! Does fanotify fix those issues?
- scaramanga 4y ago
  Lose events due to inotify buffer overflow, or provide backpressure on entire vfs and allow a watcher to DoS the entire system. Pick one. Similar "minor implementation issues" occur when considering how to fix all of the other "bugs." It's fine if you don't like the design trade-offs of a particular solution. But pretending that they aren't trade-offs, or they aren't valid choices, is just going to leave people shaking their heads at you. Edit: and by "you" i mean the authors of these articles. Although I think Marek is a very talented guy and probably just a bit hyperbolic with his titles.