3 ms·

I can't imagine there's any api-level dependency on the fd field being 8 or 16 bits. The size of the struct is going to change anyway, because it has pointers

I can't imagine there's any api-level dependency on the fd field being 8 or 16 bits. The size of the struct is going to change anyway, because it has pointers and those are going to change. Even new languages frequently rely on libc and the FILE struct, underlying whatever higher-level abstractions they have.

wahern 7y ago
Yeah, for this particular issue I don't think there's anything holding them back except the lack of demand. Especially OpenBSD, which isn't very deferential to ABI compatibility. Even on Linux opening tens of thousands or hundreds of thousands of file descriptors is exceptionally rare. For example, a couple of weeks ago I helped someone diagnose an issue with Vault which was hitting the soft descriptor resource limit. Vault apparently had no support for increasing this limit internally, neither implicitly nor explicitly (e.g. config or command-line). When I write massively concurrent applications I always increase this to the hard limit on startup (and maybe increase the hard limit if started as root, especially for a commercial product) because it's such an obvious problem which I've encountered frequently, and no support engineer or sysadmin should to be woken up at 2AM, nor users thwarted for hours or days, because of such a stupid issue, no matter that the solution is trivial (e.g. setting LimitNOFILE in the systemd unit file). So what this tells me is that despite being written in an easily scalable network programming language environment (i.e. Go), few Vault installations actually see several thousands of concurrent connections, including at Vault itself. It's a QoI issue, so if it was common I couldn't fathom an engineer not increasing the soft limit if not the hard limit on startup; and if the engineer balked out of some misplaced principle[1], surely the project or product manager would overrule them. [1] They might argue that good security practice demands obeying the inherited soft limit. But 1) almost no deployer will think about the limit, let alone the soft limit; 2) even assuming a knowledgable sysadmin, it's still not a very good excuse for not increasing it to the hard limit; and 3) a low limit makes it trivial to accidentally DoS a service.
jandrese 7y ago
It can be exposed in surprising ways, like the select() macros. That's the point of the article. You would think there would be no good reason for an application to mess with that directly, and then you change it and discover all of the applications that did so anyway.
- wahern 7y ago
  Those environments support more than SHRT_MAX descriptors in a process. It's only fopen and fdopen with this quirk. Looking at the OpenBSD source code, both fopen and fdopen explicitly check the descriptor value (which is an int, locally) and return EMFILE if it's above SHRT_MAX. See https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... and https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fdopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... Also, maybe select is a bad example as the type of struct fd_set is unrelated to the type used to store a descriptor. It's just a bitfield which can be any arbitrary integral type. On OpenBSD it's uint32_t. Perhaps you had in mind the internal array size used for the bitfield, which is a commonly encountered landmine. But that applies to select specifically, and Linux has the same problem to the same degree. poll might be a better example, but struct pollfd already uses an int for the .fd member in all those environments. pid_t and the value range of an exit status code are open issues currently causing pain not just for ABI compatibility but also API compatibility, both in practice (headaches with Linux syscalls) and with POSIX (bugs and debate on the committees). pid_t is a fairly standard issue--there are several APIs that take a pointer to pid_t, but also pid_t is conflated with int in some APIs (I forget which). Exit value is more problematic because a program can return an int-ranged value, but wait(2), waitpid(2), etc, take a pointer to int for the status argument, which encodes not only the exit value but additional metadata like the reason for exiting. Moreover, traditionally the exit value has always been encoded in 7 (or 8?) bits in the status value. These issues also bleed over into shell semantics. The next version of POSIX will address this by adding some constraints and explicit caveats to the permissible exit value range. Another interesting case is sigset_t, which must be statically sized. But arguably that's more a feature than a bug. Attempts at substantially increasing the signal number range (e.g. with real-time signals) turned out to be bad for other reasons and good examples of when dynamic limits can be bad design.

wahern 7y ago

Yeah, for this particular issue I don't think there's anything holding them back except the lack of demand. Especially OpenBSD, which isn't very deferential to ABI compatibility. Even on Linux opening tens of thousands or hundreds of thousands of file descriptors is exceptionally rare. For example, a couple of weeks ago I helped someone diagnose an issue with Vault which was hitting the soft descriptor resource limit. Vault apparently had no support for increasing this limit internally, neither implicitly nor explicitly (e.g. config or command-line). When I write massively concurrent applications I always increase this to the hard limit on startup (and maybe increase the hard limit if started as root, especially for a commercial product) because it's such an obvious problem which I've encountered frequently, and no support engineer or sysadmin should to be woken up at 2AM, nor users thwarted for hours or days, because of such a stupid issue, no matter that the solution is trivial (e.g. setting LimitNOFILE in the systemd unit file). So what this tells me is that despite being written in an easily scalable network programming language environment (i.e. Go), few Vault installations actually see several thousands of concurrent connections, including at Vault itself. It's a QoI issue, so if it was common I couldn't fathom an engineer not increasing the soft limit if not the hard limit on startup; and if the engineer balked out of some misplaced principle[1], surely the project or product manager would overrule them. [1] They might argue that good security practice demands obeying the inherited soft limit. But 1) almost no deployer will think about the limit, let alone the soft limit; 2) even assuming a knowledgable sysadmin, it's still not a very good excuse for not increasing it to the hard limit; and 3) a low limit makes it trivial to accidentally DoS a service.

jandrese 7y ago

It can be exposed in surprising ways, like the select() macros. That's the point of the article. You would think there would be no good reason for an application to mess with that directly, and then you change it and discover all of the applications that did so anyway.

wahern 7y ago
Those environments support more than SHRT_MAX descriptors in a process. It's only fopen and fdopen with this quirk. Looking at the OpenBSD source code, both fopen and fdopen explicitly check the descriptor value (which is an int, locally) and return EMFILE if it's above SHRT_MAX. See https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... and https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fdopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... Also, maybe select is a bad example as the type of struct fd_set is unrelated to the type used to store a descriptor. It's just a bitfield which can be any arbitrary integral type. On OpenBSD it's uint32_t. Perhaps you had in mind the internal array size used for the bitfield, which is a commonly encountered landmine. But that applies to select specifically, and Linux has the same problem to the same degree. poll might be a better example, but struct pollfd already uses an int for the .fd member in all those environments. pid_t and the value range of an exit status code are open issues currently causing pain not just for ABI compatibility but also API compatibility, both in practice (headaches with Linux syscalls) and with POSIX (bugs and debate on the committees). pid_t is a fairly standard issue--there are several APIs that take a pointer to pid_t, but also pid_t is conflated with int in some APIs (I forget which). Exit value is more problematic because a program can return an int-ranged value, but wait(2), waitpid(2), etc, take a pointer to int for the status argument, which encodes not only the exit value but additional metadata like the reason for exiting. Moreover, traditionally the exit value has always been encoded in 7 (or 8?) bits in the status value. These issues also bleed over into shell semantics. The next version of POSIX will address this by adding some constraints and explicit caveats to the permissible exit value range. Another interesting case is sigset_t, which must be statically sized. But arguably that's more a feature than a bug. Attempts at substantially increasing the signal number range (e.g. with real-time signals) turned out to be bad for other reasons and good examples of when dynamic limits can be bad design.