4 ms·

This is true but not entirely accurate. The use case that he's mentioning, an application like Dropbox, that relies on a good file system watcher wouldn't be to

This is true but not entirely accurate. The use case that he's mentioning, an application like Dropbox, that relies on a good file system watcher wouldn't be totally feasible with kqueue, since you're pretty much limited by the number of open file descriptors the OS allows to handle, if i don't remember wrong this number in FreeBSD is below 10000 by default, so it's limited for that use case. And that's also why OS X have FSEvents, because kqueue falls short for some use cases. I've worked on a multi-platform file system watcher ( https://bitbucket.org/SpartanJ/efsw https://bitbucket.org/SpartanJ/efsw ), and i must say that the kqueue implementation was by far the worst to work at. On the other side inotify is super simple and just works ( but sadly it's not recursive like FSEvents and IO Completion Ports ).

vezzy-fnord 12y ago
By default, yes. The kern.maxfiles sysctl value can go to the upwards of hundreds of thousands, if I recall correctly.
- throwabob214 12y ago
  Practically the limits are INT_MAX, the size of the process file descriptor table in-kernel. (And possibly a system limit on vnodes? Just guessing.) 2 billion fds makes for a lot of memory used for open files.
throwabob214 12y ago
> pretty much limited by the number of open file descriptors the OS allows to handle, if i don't remember wrong this number in FreeBSD is below 10000 by default And on Linux inotify is bound by /proc/sys/fs/inotify/max_user_watches. Six of one, half dozen the other. FreeBSD file descriptor limit is arbitrary and can be raised to INT_MAX (2 billion). That should be plenty. > i must say that the kqueue implementation was by far the worst to work at. I agree that the kqueue API is a pain to work with. I haven't tried to do a recursive file monitoring (dropbox-alike) application with it. It seems easy to implement inotify as a thin wrapper around kqueue. Maybe something to do for the FreeBSD linuxulator layer...
- SpartanJ 12y ago
  > And on Linux inotify is bound by /proc/sys/fs/inotify/max_user_watches. Six of one, half dozen the other. FreeBSD file descriptor limit is arbitrary and can be raised to INT_MAX (2 billion). That should be plenty. Yes, on Linux you're limited to max_user_watches, and you can edit that value, just like you say with kqueue... but you need root access, so for end user applications this is not really a solution. With inotify i use one watch per-folder, with kqueue you need to do it by file ( of course you can watch the folder changes, but you need to keep a copy of the file states in that folder, and then re-stat that files to find out what file was changed, not even close to an ideal solution ). So for example in my use case with inotify i can watch 65536 folders, and with kqueue less than 10000 files ( that could be a single folder! ). Edit: It seems that FreeBSD default a much bigger number for FDs ( https://news.ycombinator.com/item?id=9063893 https://news.ycombinator.com/item?id=9063893 ), so that should suffice a lot of use cases, that's good news for me!
  - throwabob214 12y ago
    > With inotify i use one watch per-folder, with kqueue you need to do it by file ( of course you can watch the folder changes, but you need to keep a copy of the file states in that folder, and then re-stat that files to find out what file was changed, not even close to an ideal solution ). So for example in my use case with inotify i can watch 65536 folders, and with kqueue less than 10000 files ( that could be a single folder! ). Ah, I looked at inotify API briefly and did not notice that directory watches return file changes. That does make it easier to use than kevent vnode watches. Mea culpa.
deleted 12y ago
[deleted]
tobik 12y ago
kern.maxfiles=782036 on FreeBSD 10.1 here and I have not changed the default at all. So it seems it is not set to 10000 by default anymore. Edit: It seems to be set dynamically depending on available memory at boot. See http://fxr.watson.org/fxr/source/kern/subr_param.c#L266 http://fxr.watson.org/fxr/source/kern/subr_param.c#L266
- SpartanJ 12y ago
  Well... that's great! :) May be it was PC-BSD, i don't have it installed right now, so i can't say it with certainty. But that number looks just enough.
GalacticDomin8r 12y ago
> if i don't remember wrong this number in FreeBSD is below 10000 by default You are wrong. ulimit -Hn; ulimit -Sn 467631 467631 Now try that on RedHat/CentOS as a default non-root user.

vezzy-fnord 12y ago

By default, yes. The kern.maxfiles sysctl value can go to the upwards of hundreds of thousands, if I recall correctly.

throwabob214 12y ago
Practically the limits are INT_MAX, the size of the process file descriptor table in-kernel. (And possibly a system limit on vnodes? Just guessing.) 2 billion fds makes for a lot of memory used for open files.

throwabob214 12y ago

> pretty much limited by the number of open file descriptors the OS allows to handle, if i don't remember wrong this number in FreeBSD is below 10000 by default And on Linux inotify is bound by /proc/sys/fs/inotify/max_user_watches. Six of one, half dozen the other. FreeBSD file descriptor limit is arbitrary and can be raised to INT_MAX (2 billion). That should be plenty. > i must say that the kqueue implementation was by far the worst to work at. I agree that the kqueue API is a pain to work with. I haven't tried to do a recursive file monitoring (dropbox-alike) application with it. It seems easy to implement inotify as a thin wrapper around kqueue. Maybe something to do for the FreeBSD linuxulator layer...

SpartanJ 12y ago
> And on Linux inotify is bound by /proc/sys/fs/inotify/max_user_watches. Six of one, half dozen the other. FreeBSD file descriptor limit is arbitrary and can be raised to INT_MAX (2 billion). That should be plenty. Yes, on Linux you're limited to max_user_watches, and you can edit that value, just like you say with kqueue... but you need root access, so for end user applications this is not really a solution. With inotify i use one watch per-folder, with kqueue you need to do it by file ( of course you can watch the folder changes, but you need to keep a copy of the file states in that folder, and then re-stat that files to find out what file was changed, not even close to an ideal solution ). So for example in my use case with inotify i can watch 65536 folders, and with kqueue less than 10000 files ( that could be a single folder! ). Edit: It seems that FreeBSD default a much bigger number for FDs ( https://news.ycombinator.com/item?id=9063893 https://news.ycombinator.com/item?id=9063893 ), so that should suffice a lot of use cases, that's good news for me!
- throwabob214 12y ago
  > With inotify i use one watch per-folder, with kqueue you need to do it by file ( of course you can watch the folder changes, but you need to keep a copy of the file states in that folder, and then re-stat that files to find out what file was changed, not even close to an ideal solution ). So for example in my use case with inotify i can watch 65536 folders, and with kqueue less than 10000 files ( that could be a single folder! ). Ah, I looked at inotify API briefly and did not notice that directory watches return file changes. That does make it easier to use than kevent vnode watches. Mea culpa.

deleted 12y ago

[deleted]

tobik 12y ago

kern.maxfiles=782036 on FreeBSD 10.1 here and I have not changed the default at all. So it seems it is not set to 10000 by default anymore. Edit: It seems to be set dynamically depending on available memory at boot. See http://fxr.watson.org/fxr/source/kern/subr_param.c#L266 http://fxr.watson.org/fxr/source/kern/subr_param.c#L266

SpartanJ 12y ago
Well... that's great! :) May be it was PC-BSD, i don't have it installed right now, so i can't say it with certainty. But that number looks just enough.

GalacticDomin8r 12y ago

> if i don't remember wrong this number in FreeBSD is below 10000 by default You are wrong. ulimit -Hn; ulimit -Sn 467631 467631 Now try that on RedHat/CentOS as a default non-root user.