5 ms·

I have two complaints about sockets: 1, bidirectional sockets probably should've been ≥2 fd's, not 1. 2, non-blocking semantics and poll/select suck. Can't b

by jchw 2mo ago

I have two complaints about sockets: 1, bidirectional sockets probably should've been ≥2 fd's, not 1. 2, non-blocking semantics and poll/select suck. Can't blame anyone in 1983 for not getting "async I/O" right since we're still struggling; at least now we have some decent answers (like io_uring on Linux.)

inigyou 2mo ago
Windows NT had overlapped IO decades before Linux. The Microsoft kernel team were alright.
- jchw 2mo ago
  Windows NT had many things decades before Linux, I've gotten used to it. There are things Windows has that I still sort of wish Linux had, I mentioned one of them just last night (RDP fast user switching/session roaming.) If we're just strictly talking about low level things, another good one would be synchronization primitives, which I guess we now have some of in Linux verbatim at this point, if only for the sake of emulation. (And of course futex2.)
  - inigyou 2mo ago
    I don't think the kernel prohibits fast user switching but you'd have to implement it at the desktop layer too. Think of ctrl-alt-f7, f8 etc. This still works today, you can run X servers as different users on different VT numbers and switch between them. All that's missing is better UI.
    - jchw 2mo ago
      The kernel definitely doesn't, although the VT layer kind of sucks and probably should've never been used to multiplex graphical sessions. (I mean, I get why it was, but oh well.) I was talking about this: https://news.ycombinator.com/item?id=49093002 https://news.ycombinator.com/item?id=49093002 And it winds up being mostly a thing you have to implement in the compositor, but none of them do so far to my knowledge. And of course the same issue applies if you wanted multiple physical seats juggling sessions between them.
      - inigyou 2mo ago
        That's systemd limitations. Who even uses systemd?
        jchw 2mo ago
        > That's systemd limitations. It's kind of the opposite, systemd-logind provides better seat management than without. I attempted to implement the same concept using libseat and seatd unsuccessfully though I do not remember exactly what I got caught on. (not in kwin but a toy compositor; I have been at this problem for a little while now.) That said, most desktop systems have only implemented systemd-logind in a limited way so far that basically just does what the VT system + display manager was already doing, that's what would need to be worked on in order to make this a reality. I was able to accomplish a prototype with only patches to Kwin, kfreerdp and plasma-login-manager, no need to patch systemd-logind or anything like that. > Who even uses systemd? Well for one thing, almost all of the major distributions; Ubuntu, Debian, Fedora/RHEL, Arch Linux, NixOS, openSuSE? and of course SteamOS now. The only major non-systemd Linux distributions I can think of are Alpine Linux and Gentoo. And Android if you wanted to count that, but I think it is special enough to be considered its own OS that just is Linux-based.
        inigyou 2mo ago
        Then it's libseat and seatd limitations. Why rely on a system that doesn't do what you want? Bypass it and DIY what you actually want.
        jchw 2mo ago
        I personally use a systemd-based distro (NixOS) and logind is able to handle this use case. The point of trying to support libseat was for the sake of non-systemd distros. (Though elogind would probably work too.) I certainly wouldn't DIY this; nothing would support my DIY version, which would defeat the purpose.
        inigyou 2mo ago
  - celiacFun 2mo ago
    Windows strengths: IRPs and fantastic async I/O, ETW events, tooling- windbg xperf driver verifier time travel debugging, RDP, doesn’t claim everything acts like a file Linux strengths: file system, open source ecosystem, ssh, lockless algorithm support
- adrian_b 2mo ago
  Mechanisms equivalent with the overlapped I/O of Windows NT (1993) already existed 30 years earlier, e.g. in IBM OS/360 and PL/I (1964/1965). The main features that were better in Windows NT than in the UNIX-derived operating systems were inherited from the DEC VAX/VMS operating system (1978) (e.g. WaitForMultipleObjects) and a part of them had been inherited from the even earlier operating system DEC RSX-11M (1974-11) (Dave Cutler also had a major role in those operating systems, so there is nothing surprising about this; Microsoft had to pay a big compensation to DEC, on the order of $ 100M, for the features that were obviously taken from the DEC operating systems). UNIX was very simplified in comparison with the operating systems that were used on bigger computers, and for some things its derivatives never caught up with those older systems, except after 2000. Only "futex" (2002) and "io_uring" (2019) have advanced the state of the art clearly beyond what already existed for AIO on the IBM mainframes around 1964/1966.
  - jchw 2mo ago
    To be fair, I think that was the entire challenge in the first place: the contemporary microcomputers were relatively limited on what they could do, so you couldn't necessarily afford to just do everything mainframes were doing. More importantly, it's not clear you would've wanted to. Having the taste and good sense to know what to "steal" and how to "steal" it was already hard enough work. A lot of good ideas took a while to come over, but a lot of probably bad ideas also never did. It's an evergreen interesting fact that Dave Cutler took perhaps a bit too much inspiration from prior VMS work to NT, but I also think there's no need to put an asterisk on it anymore than there is UNIX or anything else; it does ultimately stand on its own. Like great artists, great software architects steal.
  - pjmlp 2mo ago
    Including the use of systems programming languages with bounds checking, PL/I, PL.8, PL/S, NEWP, ...
  - celiacFun 2mo ago
    I’m not sure you can call anything in Linux/Unix as equivalent to overlapped I/O in Windows given that there aren’t any IRPs
actionfromafar 2mo ago
3, fds should have been UUIDs
- jchw 2mo ago
  fds should have been IPv6 addresses. I will not elaborate on why or how.
  - actionfromafar 2mo ago
    It's such a common problem to write to a stale file descriptor. It may be closed which is fine, at least you will have an obvious error. But what's much worse is when the stale descriptor actually works, because something else in the program opened a new fd and your stale descriptor is not stale anymore, it just points at something unexpected. Opaque handles are often a blessing. Maybe UUID would be overkill (it certainly would have been back in the day of 16-bit unix machines) but something fairly large would have helped a lot. Even just incrementing and rolling over a 16-bit value would have helped, instead of situations such as closing stdout and stderr, opening a file, and now random error logs sprinkle into your PDF or whatever. Yes, "you are holding it wrong", but it should be hard to "hold it wrong", not easy.
    - jchw 2mo ago
      Joking aside I pretty much agree with this. Windows NT Object handles work pretty well. I assume they are pointers of some kind under the hood, but either way a machine word is probably large enough as it seems unlikely you would ever need to handle more open handles than you have bytes of addressable memory.
zbentley 2mo ago
> non-blocking semantics and poll/select suck. I'd argue that it's not that the semantics suck, it's that there are too many of them. We have O_NONBLOCK, multiple multiplexers (for sane reasons--I don't begrudge 1980s folks for not thinking about fd set size and copy overhead for select(2) either), and others. What's worse, they don't all work with all FDs--not only are regular files not nonblock-able in the same way that sockets are, but all sorts of other FD-exposed capabilities (signalfds, timerfds, memfds, pidfds) do or don't support nonblocking semantics and multiplexing in all sorts of weird ways. If those old system designers had stuck with keeping the async IO syscall space small (e.g. "you only get select/poll" or "you only get read(fds) and read_noblock(fds, timeout)") and consistent (by drawing a hard line at "if you expose something as an FD, it must support all APIs that generically handle FDs"), we would have ended up in a better place. Sure, that would have slowed down some implementations (e.g. vfs drivers), but would also have massively sped up development against a lot of these APIs. Ah, well, hindsight is 20/20 I guess.
- adrian_b 2mo ago
  Allready in 1983, 4.2BSD had 3 different attempts at doing multiplexed/asynchronous I/O: Non-blocking I/O: "O_NONBLOCK" with "EWOULDBLOCK" (or "EAGAIN") Signal-driven I/O: "O_ASYNC" with "SIGIO" Synchronous I/O multiplexing: "select()" All 3 had various defects, especially with a large number of concurrent I/O actions. In UNIX-derived operating systems, after 1983 there have been many other attempts to implement something better than these 3 (starting with System V "poll" and with POSIX AIO, and then with various incompatible approaches in Solaris, FreeBSD and Linux), but none were good enough and most were seriously inferior to methods of doing asynchronous I/O that existed in some IBM and DEC operating systems decades earlier. In my opinion, only io_uring has finally solved the problem of multiplex asynchronous I/O in Linux, and in a manner much better than in all older operating systems. I consider all the many older alternatives that exist for liburing as obsolete.
  - sscaryterry 2mo ago
    This. I'd break-out in a cold sweat if I had to work on this again.
celiacFun 2mo ago
I just can’t understand why the Linux kernel has avoided tracking completions for so long and has such hard time with async I/O. Windows has had IRPs and overlapped I/O for decades, asyc is the default and there just isn’t much reason to ever use synchronous I/O.

by jchw 2mo ago

inigyou 2mo ago

Windows NT had overlapped IO decades before Linux. The Microsoft kernel team were alright.

jchw 2mo ago
Windows NT had many things decades before Linux, I've gotten used to it. There are things Windows has that I still sort of wish Linux had, I mentioned one of them just last night (RDP fast user switching/session roaming.) If we're just strictly talking about low level things, another good one would be synchronization primitives, which I guess we now have some of in Linux verbatim at this point, if only for the sake of emulation. (And of course futex2.)
- inigyou 2mo ago
  I don't think the kernel prohibits fast user switching but you'd have to implement it at the desktop layer too. Think of ctrl-alt-f7, f8 etc. This still works today, you can run X servers as different users on different VT numbers and switch between them. All that's missing is better UI.
  - jchw 2mo ago
    The kernel definitely doesn't, although the VT layer kind of sucks and probably should've never been used to multiplex graphical sessions. (I mean, I get why it was, but oh well.) I was talking about this: https://news.ycombinator.com/item?id=49093002 https://news.ycombinator.com/item?id=49093002 And it winds up being mostly a thing you have to implement in the compositor, but none of them do so far to my knowledge. And of course the same issue applies if you wanted multiple physical seats juggling sessions between them.
    - inigyou 2mo ago
      That's systemd limitations. Who even uses systemd?
      - jchw 2mo ago
        > That's systemd limitations. It's kind of the opposite, systemd-logind provides better seat management than without. I attempted to implement the same concept using libseat and seatd unsuccessfully though I do not remember exactly what I got caught on. (not in kwin but a toy compositor; I have been at this problem for a little while now.) That said, most desktop systems have only implemented systemd-logind in a limited way so far that basically just does what the VT system + display manager was already doing, that's what would need to be worked on in order to make this a reality. I was able to accomplish a prototype with only patches to Kwin, kfreerdp and plasma-login-manager, no need to patch systemd-logind or anything like that. > Who even uses systemd? Well for one thing, almost all of the major distributions; Ubuntu, Debian, Fedora/RHEL, Arch Linux, NixOS, openSuSE? and of course SteamOS now. The only major non-systemd Linux distributions I can think of are Alpine Linux and Gentoo. And Android if you wanted to count that, but I think it is special enough to be considered its own OS that just is Linux-based.
        inigyou 2mo ago
        Then it's libseat and seatd limitations. Why rely on a system that doesn't do what you want? Bypass it and DIY what you actually want.
        jchw 2mo ago
        I personally use a systemd-based distro (NixOS) and logind is able to handle this use case. The point of trying to support libseat was for the sake of non-systemd distros. (Though elogind would probably work too.) I certainly wouldn't DIY this; nothing would support my DIY version, which would defeat the purpose.
        inigyou 2mo ago
- celiacFun 2mo ago
  Windows strengths: IRPs and fantastic async I/O, ETW events, tooling- windbg xperf driver verifier time travel debugging, RDP, doesn’t claim everything acts like a file Linux strengths: file system, open source ecosystem, ssh, lockless algorithm support
adrian_b 2mo ago
Mechanisms equivalent with the overlapped I/O of Windows NT (1993) already existed 30 years earlier, e.g. in IBM OS/360 and PL/I (1964/1965). The main features that were better in Windows NT than in the UNIX-derived operating systems were inherited from the DEC VAX/VMS operating system (1978) (e.g. WaitForMultipleObjects) and a part of them had been inherited from the even earlier operating system DEC RSX-11M (1974-11) (Dave Cutler also had a major role in those operating systems, so there is nothing surprising about this; Microsoft had to pay a big compensation to DEC, on the order of $ 100M, for the features that were obviously taken from the DEC operating systems). UNIX was very simplified in comparison with the operating systems that were used on bigger computers, and for some things its derivatives never caught up with those older systems, except after 2000. Only "futex" (2002) and "io_uring" (2019) have advanced the state of the art clearly beyond what already existed for AIO on the IBM mainframes around 1964/1966.
- jchw 2mo ago
  To be fair, I think that was the entire challenge in the first place: the contemporary microcomputers were relatively limited on what they could do, so you couldn't necessarily afford to just do everything mainframes were doing. More importantly, it's not clear you would've wanted to. Having the taste and good sense to know what to "steal" and how to "steal" it was already hard enough work. A lot of good ideas took a while to come over, but a lot of probably bad ideas also never did. It's an evergreen interesting fact that Dave Cutler took perhaps a bit too much inspiration from prior VMS work to NT, but I also think there's no need to put an asterisk on it anymore than there is UNIX or anything else; it does ultimately stand on its own. Like great artists, great software architects steal.
- pjmlp 2mo ago
  Including the use of systems programming languages with bounds checking, PL/I, PL.8, PL/S, NEWP, ...
- celiacFun 2mo ago
  I’m not sure you can call anything in Linux/Unix as equivalent to overlapped I/O in Windows given that there aren’t any IRPs

actionfromafar 2mo ago

3, fds should have been UUIDs

jchw 2mo ago
fds should have been IPv6 addresses. I will not elaborate on why or how.
- actionfromafar 2mo ago
  It's such a common problem to write to a stale file descriptor. It may be closed which is fine, at least you will have an obvious error. But what's much worse is when the stale descriptor actually works, because something else in the program opened a new fd and your stale descriptor is not stale anymore, it just points at something unexpected. Opaque handles are often a blessing. Maybe UUID would be overkill (it certainly would have been back in the day of 16-bit unix machines) but something fairly large would have helped a lot. Even just incrementing and rolling over a 16-bit value would have helped, instead of situations such as closing stdout and stderr, opening a file, and now random error logs sprinkle into your PDF or whatever. Yes, "you are holding it wrong", but it should be hard to "hold it wrong", not easy.
  - jchw 2mo ago
    Joking aside I pretty much agree with this. Windows NT Object handles work pretty well. I assume they are pointers of some kind under the hood, but either way a machine word is probably large enough as it seems unlikely you would ever need to handle more open handles than you have bytes of addressable memory.

zbentley 2mo ago

> non-blocking semantics and poll/select suck. I'd argue that it's not that the semantics suck, it's that there are too many of them. We have O_NONBLOCK, multiple multiplexers (for sane reasons--I don't begrudge 1980s folks for not thinking about fd set size and copy overhead for select(2) either), and others. What's worse, they don't all work with all FDs--not only are regular files not nonblock-able in the same way that sockets are, but all sorts of other FD-exposed capabilities (signalfds, timerfds, memfds, pidfds) do or don't support nonblocking semantics and multiplexing in all sorts of weird ways. If those old system designers had stuck with keeping the async IO syscall space small (e.g. "you only get select/poll" or "you only get read(fds) and read_noblock(fds, timeout)") and consistent (by drawing a hard line at "if you expose something as an FD, it must support all APIs that generically handle FDs"), we would have ended up in a better place. Sure, that would have slowed down some implementations (e.g. vfs drivers), but would also have massively sped up development against a lot of these APIs. Ah, well, hindsight is 20/20 I guess.

adrian_b 2mo ago
Allready in 1983, 4.2BSD had 3 different attempts at doing multiplexed/asynchronous I/O: Non-blocking I/O: "O_NONBLOCK" with "EWOULDBLOCK" (or "EAGAIN") Signal-driven I/O: "O_ASYNC" with "SIGIO" Synchronous I/O multiplexing: "select()" All 3 had various defects, especially with a large number of concurrent I/O actions. In UNIX-derived operating systems, after 1983 there have been many other attempts to implement something better than these 3 (starting with System V "poll" and with POSIX AIO, and then with various incompatible approaches in Solaris, FreeBSD and Linux), but none were good enough and most were seriously inferior to methods of doing asynchronous I/O that existed in some IBM and DEC operating systems decades earlier. In my opinion, only io_uring has finally solved the problem of multiplex asynchronous I/O in Linux, and in a manner much better than in all older operating systems. I consider all the many older alternatives that exist for liburing as obsolete.
- sscaryterry 2mo ago
  This. I'd break-out in a cold sweat if I had to work on this again.

celiacFun 2mo ago

I just can’t understand why the Linux kernel has avoided tracking completions for so long and has such hard time with async I/O. Windows has had IRPs and overlapped I/O for decades, asyc is the default and there just isn’t much reason to ever use synchronous I/O.