4 ms·

The default limit is in place because select() uses a fixed size bitmap for file descriptors. If a program that uses select opens more than that many files it w

The default limit is in place because select() uses a fixed size bitmap for file descriptors. If a program that uses select opens more than that many files it will make an out of bounds write. It is probably better to make the file descriptor allocation fail than have memory corruption. All programs that don't use select() should raise the limit to the hard limit on startup. Then drop it back down before exec()ing another program. It is a silly dance but that is the cost of legacy. I'm surprised that these programs that are affected can't add this dance in.

crest 3y ago
Select has 1024 bit/file descriptor bitmap and anyone using it gets what they deserve. MacOS also provides poll() as a slightly less braindead, but still POSIX option. The proper solution is to use kqueue()/kevent() allowing userspace processes to efficiently get notified about changes to tens of thousands of file descriptors (tracking a million file descriptors using this API on FreeBSD works fine).
- kevincox 3y ago
  I don't know if I would call anyone unfortunate enough to not know about the limitations of select() deserving of such limitations. At least on my Linux box the man page does have a warning at the top of the description section but it is easy to accidentally skim over. It would be interesting to do something like avoid defining that symbol by default, require `-DENABLE_OBSOLETE_SELECT_API` to make it available. It would cause trouble for compiling old software but it is easy to remedy and at least makes new users extra aware that they shouldn't start using this function.
  - avhception 3y ago
    Unfortunately, adding in an option like `-DENABLE_OBSOLETE_SELECT_API` is only trivial if you have a good understanding of the build system in use and sometimes other nuts and bolts of a given project. Tracking down a build failure in some old software project that may or may not have approachable people left on the team through elaborate build scripts that may or may not surface the error in a way that may or may not be easily comprehensible for an outsider can be really hard.
  - rollcat 3y ago
    Apropos Macs and man pages, does anyone know if there's any up-to-date website that allows browsing them online? Like https://man7.org/linux/man-pages/ https://man7.org/linux/man-pages/ or https://man.openbsd.org https://man.openbsd.org, ideally per release and per architecture.
    - throw0101a 3y ago
      * https://keith.github.io/xcode-man-pages/ https://keith.github.io/xcode-man-pages/ * https://github.com/keith/xcode-man-pages https://github.com/keith/xcode-man-pages ?
  - pjmlp 3y ago
    I expect anyone coding UNIX systems to have more sources than only man pages, like classic UNIX books.
- mort96 3y ago
  I like using poll(). I don't generally write software which waits for tens of thousands of FDs, I wait for a couple. Sacrificing platform support for more or less nothing in return doesn't make a lot of sense. Kqueue/kevent/epoll are good options if you have very particular needs, such as a huge number of file descriptors, but I'd argue poll() should be the go-to.
  - arghwhat 3y ago
    You get kqueue and epoll for free with most small event loop libraries. But yes, where you only need to poll a few things, poll(3) is fine.
    - mort96 3y ago
      I guess if you're using an event loop library you don't have to think about it. For people who make event loop libraries, I agree that they should probably support kqueue/kevent+epoll. I was talking about normal application code that's not using such an event loop library.
StewardMcOy 3y ago
This seems like an unfortunate, but reasonable, given the circumstances, approach, but you'd have to make sure that none of your dependencies, including Apple's frameworks, have code that calls select.
saagarjha 3y ago
Are you sure it's that, and not people setting ulimit to something very low and causing critical programs to start failing in ways that open up security issues?
- kevincox 3y ago
  It could be both. But having a failure be a security issue is a fail-open design and should be avoided in most cases. Having out-of-bounds memory writes can be exploited in a variety of ways and can provide exploits that affect even fail-closed designs.
arghwhat 3y ago
I am fairly confident that this is not the reason for the file descriptor limit, especially since select was superseded over 3 decades ago by by poll (1987, picked up by Linux in 1997), which in turn is superseded locally by kqueue. Use of select in modern times is a bug. ulimits are quotas that disallow excessive resource consumption by an application, not bug shields.
hashhar 3y ago
The reasonable solution for that would be to fail calls to select() if more than FD_SETSIZE fds are being held instead of nerfing all applications to do the setrlimit dance - some of which may not be actively maintained or even if they are would take time for fixes to be available and distributed.
- kevincox 3y ago
  The problem is that the memory corruption occurs when preparing the arguments to `select()` so by the time `select` is called it is already too late. Having select abort the program could make it harder to exploit as the corruption likely occurs on the caller's stack but doesn't completely solve the problem. I guess the real solution would be updating `FD_SET()` and `FD_CLR()` to abort if `fd > FD_SETSIZE`. IDK if writing to the fd set outside of these two functions is officially supported.

crest 3y ago

Select has 1024 bit/file descriptor bitmap and anyone using it gets what they deserve. MacOS also provides poll() as a slightly less braindead, but still POSIX option. The proper solution is to use kqueue()/kevent() allowing userspace processes to efficiently get notified about changes to tens of thousands of file descriptors (tracking a million file descriptors using this API on FreeBSD works fine).

kevincox 3y ago
I don't know if I would call anyone unfortunate enough to not know about the limitations of select() deserving of such limitations. At least on my Linux box the man page does have a warning at the top of the description section but it is easy to accidentally skim over. It would be interesting to do something like avoid defining that symbol by default, require `-DENABLE_OBSOLETE_SELECT_API` to make it available. It would cause trouble for compiling old software but it is easy to remedy and at least makes new users extra aware that they shouldn't start using this function.
- avhception 3y ago
  Unfortunately, adding in an option like `-DENABLE_OBSOLETE_SELECT_API` is only trivial if you have a good understanding of the build system in use and sometimes other nuts and bolts of a given project. Tracking down a build failure in some old software project that may or may not have approachable people left on the team through elaborate build scripts that may or may not surface the error in a way that may or may not be easily comprehensible for an outsider can be really hard.
- rollcat 3y ago
  Apropos Macs and man pages, does anyone know if there's any up-to-date website that allows browsing them online? Like https://man7.org/linux/man-pages/ https://man7.org/linux/man-pages/ or https://man.openbsd.org https://man.openbsd.org, ideally per release and per architecture.
  - throw0101a 3y ago
    * https://keith.github.io/xcode-man-pages/ https://keith.github.io/xcode-man-pages/ * https://github.com/keith/xcode-man-pages https://github.com/keith/xcode-man-pages ?
- pjmlp 3y ago
  I expect anyone coding UNIX systems to have more sources than only man pages, like classic UNIX books.
mort96 3y ago
I like using poll(). I don't generally write software which waits for tens of thousands of FDs, I wait for a couple. Sacrificing platform support for more or less nothing in return doesn't make a lot of sense. Kqueue/kevent/epoll are good options if you have very particular needs, such as a huge number of file descriptors, but I'd argue poll() should be the go-to.
- arghwhat 3y ago
  You get kqueue and epoll for free with most small event loop libraries. But yes, where you only need to poll a few things, poll(3) is fine.
  - mort96 3y ago
    I guess if you're using an event loop library you don't have to think about it. For people who make event loop libraries, I agree that they should probably support kqueue/kevent+epoll. I was talking about normal application code that's not using such an event loop library.

StewardMcOy 3y ago

This seems like an unfortunate, but reasonable, given the circumstances, approach, but you'd have to make sure that none of your dependencies, including Apple's frameworks, have code that calls select.

saagarjha 3y ago

Are you sure it's that, and not people setting ulimit to something very low and causing critical programs to start failing in ways that open up security issues?

kevincox 3y ago
It could be both. But having a failure be a security issue is a fail-open design and should be avoided in most cases. Having out-of-bounds memory writes can be exploited in a variety of ways and can provide exploits that affect even fail-closed designs.

arghwhat 3y ago

I am fairly confident that this is not the reason for the file descriptor limit, especially since select was superseded over 3 decades ago by by poll (1987, picked up by Linux in 1997), which in turn is superseded locally by kqueue. Use of select in modern times is a bug. ulimits are quotas that disallow excessive resource consumption by an application, not bug shields.

hashhar 3y ago

The reasonable solution for that would be to fail calls to select() if more than FD_SETSIZE fds are being held instead of nerfing all applications to do the setrlimit dance - some of which may not be actively maintained or even if they are would take time for fixes to be available and distributed.

kevincox 3y ago
The problem is that the memory corruption occurs when preparing the arguments to `select()` so by the time `select` is called it is already too late. Having select abort the program could make it harder to exploit as the corruption likely occurs on the caller's stack but doesn't completely solve the problem. I guess the real solution would be updating `FD_SET()` and `FD_CLR()` to abort if `fd > FD_SETSIZE`. IDK if writing to the fd set outside of these two functions is officially supported.