4 ms·

Lessons from the Unix Stdio ABI: 40 Years Later

51 points by polm23 7y ago

moonchild 7y ago
> FreeBSD, NetBSD, and OpenBSD today are all still limited to a short, which on current mainstay processors is a signed 16-bit quantity or 32767 Why did they not take advantage of the move to 64-bit to change that? (Or, at the very least, more recent architecture ports like riscv.)
- jandrese 7y ago
  Some of that is the developers being practical. If they went through and updated all of the ABIs it would take years (it took years to update just time_t to 64 bits!) and people would be constantly hounding them to just let them use their 8GB of memory. Stdio is just one of thousands of cases where decisions from the 70s no longer make sense on modern machines. Plus every change you make is going to break some app somewhere, so if you change everything at once it causes chaos in the userland. This is one reason why developing new languages is so appealing. You don't have to worry about breaking old code or maintaining workarounds for long-extinct machine architectures. Of course that's ruined the first time someone writes something in your new language.
  - compiler-guy 7y ago
    Once you have existing code, the decision to break things is very hard. This exact history in early C is why even Java has an unintuitive precedence around & vs &&: "After all, we had several hundred kilobytes of source code, and maybe 3 installations...." https://www.lysator.liu.se/c/dmr-on-or.html https://www.lysator.liu.se/c/dmr-on-or.html
  - moonchild 7y ago
    I can't imagine there's any api-level dependency on the fd field being 8 or 16 bits. The size of the struct is going to change anyway, because it has pointers and those are going to change. Even new languages frequently rely on libc and the FILE struct, underlying whatever higher-level abstractions they have.
    - wahern 7y ago
      Yeah, for this particular issue I don't think there's anything holding them back except the lack of demand. Especially OpenBSD, which isn't very deferential to ABI compatibility. Even on Linux opening tens of thousands or hundreds of thousands of file descriptors is exceptionally rare. For example, a couple of weeks ago I helped someone diagnose an issue with Vault which was hitting the soft descriptor resource limit. Vault apparently had no support for increasing this limit internally, neither implicitly nor explicitly (e.g. config or command-line). When I write massively concurrent applications I always increase this to the hard limit on startup (and maybe increase the hard limit if started as root, especially for a commercial product) because it's such an obvious problem which I've encountered frequently, and no support engineer or sysadmin should to be woken up at 2AM, nor users thwarted for hours or days, because of such a stupid issue, no matter that the solution is trivial (e.g. setting LimitNOFILE in the systemd unit file). So what this tells me is that despite being written in an easily scalable network programming language environment (i.e. Go), few Vault installations actually see several thousands of concurrent connections, including at Vault itself. It's a QoI issue, so if it was common I couldn't fathom an engineer not increasing the soft limit if not the hard limit on startup; and if the engineer balked out of some misplaced principle[1], surely the project or product manager would overrule them. [1] They might argue that good security practice demands obeying the inherited soft limit. But 1) almost no deployer will think about the limit, let alone the soft limit; 2) even assuming a knowledgable sysadmin, it's still not a very good excuse for not increasing it to the hard limit; and 3) a low limit makes it trivial to accidentally DoS a service.
    - jandrese 7y ago
      It can be exposed in surprising ways, like the select() macros. That's the point of the article. You would think there would be no good reason for an application to mess with that directly, and then you change it and discover all of the applications that did so anyway.
      - wahern 7y ago
        Those environments support more than SHRT_MAX descriptors in a process. It's only fopen and fdopen with this quirk. Looking at the OpenBSD source code, both fopen and fdopen explicitly check the descriptor value (which is an int, locally) and return EMFILE if it's above SHRT_MAX. See https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... and https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fdopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... Also, maybe select is a bad example as the type of struct fd_set is unrelated to the type used to store a descriptor. It's just a bitfield which can be any arbitrary integral type. On OpenBSD it's uint32_t. Perhaps you had in mind the internal array size used for the bitfield, which is a commonly encountered landmine. But that applies to select specifically, and Linux has the same problem to the same degree. poll might be a better example, but struct pollfd already uses an int for the .fd member in all those environments. pid_t and the value range of an exit status code are open issues currently causing pain not just for ABI compatibility but also API compatibility, both in practice (headaches with Linux syscalls) and with POSIX (bugs and debate on the committees). pid_t is a fairly standard issue--there are several APIs that take a pointer to pid_t, but also pid_t is conflated with int in some APIs (I forget which). Exit value is more problematic because a program can return an int-ranged value, but wait(2), waitpid(2), etc, take a pointer to int for the status argument, which encodes not only the exit value but additional metadata like the reason for exiting. Moreover, traditionally the exit value has always been encoded in 7 (or 8?) bits in the status value. These issues also bleed over into shell semantics. The next version of POSIX will address this by adding some constraints and explicit caveats to the permissible exit value range. Another interesting case is sigset_t, which must be statically sized. But arguably that's more a feature than a bug. Attempts at substantially increasing the signal number range (e.g. with real-time signals) turned out to be bad for other reasons and good examples of when dynamic limits can be bad design.
bogomipz 7y ago
The article states: >"Here, while there are a number of members which have the same types, the layout of these such as the order of the members is rather different. There’s nothing wrong with that; however, it’s pieces like this that are part of each system’s ABI." The author linked to the System V ABI and the processor-specific ABI docs and in the "Further Reading" section has links to the stdio header files. Are the system ABIs that are not covered by the System V ABI and the processor-specific ABIs always defined in header files then? There's no singular or centralized spec?
hootbootscoot 7y ago
This is a funny and informative piece.

51 points by polm23 7y ago

> FreeBSD, NetBSD, and OpenBSD today are all still limited to a short, which on current mainstay processors is a signed 16-bit quantity or 32767 Why did they not take advantage of the move to 64-bit to change that? (Or, at the very least, more recent architecture ports like riscv.)

jandrese 7y ago
Some of that is the developers being practical. If they went through and updated all of the ABIs it would take years (it took years to update just time_t to 64 bits!) and people would be constantly hounding them to just let them use their 8GB of memory. Stdio is just one of thousands of cases where decisions from the 70s no longer make sense on modern machines. Plus every change you make is going to break some app somewhere, so if you change everything at once it causes chaos in the userland. This is one reason why developing new languages is so appealing. You don't have to worry about breaking old code or maintaining workarounds for long-extinct machine architectures. Of course that's ruined the first time someone writes something in your new language.
- compiler-guy 7y ago
  Once you have existing code, the decision to break things is very hard. This exact history in early C is why even Java has an unintuitive precedence around & vs &&: "After all, we had several hundred kilobytes of source code, and maybe 3 installations...." https://www.lysator.liu.se/c/dmr-on-or.html https://www.lysator.liu.se/c/dmr-on-or.html
- moonchild 7y ago
  I can't imagine there's any api-level dependency on the fd field being 8 or 16 bits. The size of the struct is going to change anyway, because it has pointers and those are going to change. Even new languages frequently rely on libc and the FILE struct, underlying whatever higher-level abstractions they have.
  - wahern 7y ago
    Yeah, for this particular issue I don't think there's anything holding them back except the lack of demand. Especially OpenBSD, which isn't very deferential to ABI compatibility. Even on Linux opening tens of thousands or hundreds of thousands of file descriptors is exceptionally rare. For example, a couple of weeks ago I helped someone diagnose an issue with Vault which was hitting the soft descriptor resource limit. Vault apparently had no support for increasing this limit internally, neither implicitly nor explicitly (e.g. config or command-line). When I write massively concurrent applications I always increase this to the hard limit on startup (and maybe increase the hard limit if started as root, especially for a commercial product) because it's such an obvious problem which I've encountered frequently, and no support engineer or sysadmin should to be woken up at 2AM, nor users thwarted for hours or days, because of such a stupid issue, no matter that the solution is trivial (e.g. setting LimitNOFILE in the systemd unit file). So what this tells me is that despite being written in an easily scalable network programming language environment (i.e. Go), few Vault installations actually see several thousands of concurrent connections, including at Vault itself. It's a QoI issue, so if it was common I couldn't fathom an engineer not increasing the soft limit if not the hard limit on startup; and if the engineer balked out of some misplaced principle[1], surely the project or product manager would overrule them. [1] They might argue that good security practice demands obeying the inherited soft limit. But 1) almost no deployer will think about the limit, let alone the soft limit; 2) even assuming a knowledgable sysadmin, it's still not a very good excuse for not increasing it to the hard limit; and 3) a low limit makes it trivial to accidentally DoS a service.
  - jandrese 7y ago
    It can be exposed in surprising ways, like the select() macros. That's the point of the article. You would think there would be no good reason for an application to mess with that directly, and then you change it and discover all of the applications that did so anyway.
    - wahern 7y ago
      Those environments support more than SHRT_MAX descriptors in a process. It's only fopen and fdopen with this quirk. Looking at the OpenBSD source code, both fopen and fdopen explicitly check the descriptor value (which is an int, locally) and return EMFILE if it's above SHRT_MAX. See https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... and https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio/fdopen.c.diff?r1=1.5&r2=1.6 https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/lib/libc/stdio... Also, maybe select is a bad example as the type of struct fd_set is unrelated to the type used to store a descriptor. It's just a bitfield which can be any arbitrary integral type. On OpenBSD it's uint32_t. Perhaps you had in mind the internal array size used for the bitfield, which is a commonly encountered landmine. But that applies to select specifically, and Linux has the same problem to the same degree. poll might be a better example, but struct pollfd already uses an int for the .fd member in all those environments. pid_t and the value range of an exit status code are open issues currently causing pain not just for ABI compatibility but also API compatibility, both in practice (headaches with Linux syscalls) and with POSIX (bugs and debate on the committees). pid_t is a fairly standard issue--there are several APIs that take a pointer to pid_t, but also pid_t is conflated with int in some APIs (I forget which). Exit value is more problematic because a program can return an int-ranged value, but wait(2), waitpid(2), etc, take a pointer to int for the status argument, which encodes not only the exit value but additional metadata like the reason for exiting. Moreover, traditionally the exit value has always been encoded in 7 (or 8?) bits in the status value. These issues also bleed over into shell semantics. The next version of POSIX will address this by adding some constraints and explicit caveats to the permissible exit value range. Another interesting case is sigset_t, which must be statically sized. But arguably that's more a feature than a bug. Attempts at substantially increasing the signal number range (e.g. with real-time signals) turned out to be bad for other reasons and good examples of when dynamic limits can be bad design.

bogomipz 7y ago

The article states: >"Here, while there are a number of members which have the same types, the layout of these such as the order of the members is rather different. There’s nothing wrong with that; however, it’s pieces like this that are part of each system’s ABI." The author linked to the System V ABI and the processor-specific ABI docs and in the "Further Reading" section has links to the stdio header files. Are the system ABIs that are not covered by the System V ABI and the processor-specific ABIs always defined in header files then? There's no singular or centralized spec?

hootbootscoot 7y ago

This is a funny and informative piece.