4 ms·

While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? Where is the tech debt? In the kernel only? Does it aff

by klabb3 2y ago

While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? Where is the tech debt? In the kernel only? Does it affect user space? For who is it more or less composable? I personally think ”async IO” (a term I don’t love, but anyway), should be agreed upon and settled across OSs. Not necessarily exact syscall equivalence, but the major design choices needs to be the same. For instance, who owns the buffers (kernel or user), and should it be completion or ready-based event systems? The reason I think this is so important to agree upon, is that language runtimes (and other custom runtimes like libuv) need to be simplified in order to allow for healthy growth. Any language that doesn’t have cross-platform support for ”async IO” is completely nerfed, increasing the barrier of entry for things that otherwise would be ”graduate level” projects, into deep OS/kernel knowledge that is hard to find outside FAANG and a few other spaces.

BobbyTables2 2y ago
I fully agree. Synchronous I/O looks appealing for beginners and simple projects. It seems like the simplest path for things where I/O isn’t the primary focus. Then one suddenly finds themselves juggling ephemeral threads to “background” tasks or such. The application quickly becomes a multi-threaded monster and still suffers responsiveness issues from synchronous use. The Tower of Babel created from third party runtimes further adds to the mess even for those who try to do better.
- worik 2y ago
  > Synchronous I/O looks appealing for beginners and simple projects. Yes. And. Synchronous I/O is suitable for almost all projects When it is unsuitable it is catastrophicly unsuitable,
  - astrange 2y ago
    Synchronous I/O is actually better than async for many programs, partly because almost all async systems were designed without thinking about priority inversions and can't resolve them properly. If you have futures, you have this problem.
    - astrange 2y ago
      Forgot to say, the correctly designed ones are called "structured concurrency".
    - p_ing 2y ago
      Since the NT kernel is async throughout, it solves this with periodic priority boosts for non-running but runnable threads. I swear, the kernel designers just need to look at NT :-P But we're well beyond a complete rewrite of old kernels, sadly.
      - astrange 2y ago
        That can work but I think it's cheating. Especially if your scheduling is tied into power management or you have high/low power cores, then boosting the low priority threads will waste power.
- liontwist 2y ago
  The entire unix operating system is designed to abstract away async problems and turn them sync so you can write sane code (if, else, for). In my experience simple blocking code is incredibly misunderstood. Most programmers stop their thinking at “blocking is slow and non blocking is fast!”
  - mschuster91 2y ago
    > The entire unix operating system is designed to abstract away async problems and turn them sync so you can write sane code (if, else, for). That only works for workloads that can be independently parallelized. For stuff that cannot be - which is the majority of consumer workload - you're in for a world of pain with locking and whatnot other issues.
    - liontwist 2y ago
      No. It’s almost all the time that your computer has more than one thing to do; - your browser uses a process per tab - your web server responds to multiple requests in parallel - your web app is going to launch other programs - your desktop ui is going to have a compositing server separate from each application - your compiler is going to work on multiple source files
  - mzimbres 2y ago
    > In my experience simple blocking code is incredibly misunderstood. I don't think so. You can't even implement a full-duplex protocol properly with sync IO since reading and writing can't happen at the same time on the same thread. Splitting them to different threads also won't help since accessing shared state requires synchronization with e.g. mutex that kills again simultaneous reading and writing.
    - liontwist 2y ago
      How do you propose to implement duplex IO? And what’s the example case? With threads the reading and writing can happen simultaneously. The synchronization is only on whatever data you need to share. Note that if reading and writing are dependent, then single thread actually makes sense because you can’t do one without the other. Otherwise the pattern of logic you will execute for reading or writing is completely different, which is why threading is the perfect abstraction.
jandrewrogers 2y ago
The differences in design reflect differences in use case assumptions, which in turn reflects that different OS tend to be used for different workloads. Broadly speaking it is difficult to use async I/O well without understanding the implementation details. It is a sharp tool for experts, not magic “go faster” juice.
- vacuity 2y ago
  Which is why parent is calling for standardization, so that people don't usually need to be experts. I reject that there are a diverse range of async abstractions that are all justified such that a given OS only supports some well. More likely, that means the designs are poorly done. Although I can't say anything if it's for backwards compatibility, how I loathe and crave it.
  - jandrewrogers 2y ago
    The bigger issue is that no OS was designed for expansive and flexible async functionality. The implementations reflect practical limitations of remaining compatible with the rest of the guarantees the kernel makes. I would love a portable standard, I’ve been doing async for a long time and I would benefit greatly, but I also understand why it hasn’t happened. A good standard would likely require every OS to materially redesign its internals in a way that would have backward compatibility issues for the other users of the system. I am not against the idea since I would clearly benefit personally, but I understand why the tradeoffs make it unlikely to happen. Even going from io_submit to io_uring just on Linux changes the way you need to architect your software, aside from API changes, and that is the same kernel. It deeply touches the behavior of the system.
    - p_ing 2y ago
      > The bigger issue is that no OS was designed for expansive and flexible async functionality. The NT kernel was designed in this very way. All I/O in the NT kernel is async. It was one of the design principles.
bluetomcat 2y ago
The claim is silly and unfounded. If anything, kqueue is the interface that pollutes userspace less. You have a single kevent call that is used both for waiting and registering events represented by (filter, ident) tuples. All of the data related to the event is contained in a struct that’s passed between the kernel and the user. For non-fd events (EVFILT_PROC, EVFILT_TIMER, EVFILT_SIGNAL), it is much more straightforward to use compared to the Linux way where you need to keep track of one more resource, read specific binary data from it, have specific flags, etc.
- somat 2y ago
  The article reads more like pr damage control than an actual complaint. epoll has a reputation as a fragile, hard to correctly use interface. and kqueue as relatively simple and sane. So the question always is. why didn't linux just adopt the already existing kqueue instead? Sometimes it seems that linux does end up with more than it's fair share of technical excellence coupled to a bad interface. iptables, epoll and git come to mind.
  - adrian_b 2y ago
    This other article has been discussed in the past on HN: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/ https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-... Because it discusses epoll in much more detail, it is far more convincing than the parent article of this thread. The conclusion of that article is that how to use correctly epoll is not at all obvious and it has some pitfalls that are not easy to avoid. Therefore epoll seems to be more affected by a serious technical debt, i.e. by a less than good design of the original API.
Aurornis 2y ago
> While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? I came to the comments hoping to get more explanation. I was waiting for the full explanation of the technical debt but then the article just came to an abrupt end. I wonder if the article started as a short history and explanation of differences, but then the dramatic headline was chosen to drive more clicks? The article could have been interesting by itself without setting up the “mountain of technical debt” claim in the headline.
- mst 2y ago
  I know ariadne well enough that I'm pretty confident that the headline will have been chosen out of annoyance rather than click maximisation. That doesn't mean I agree with the conclusion (I am ambivalent and would have to think rather more before having an opinion myself) but I'm reasonably sure of the motivation nonetheless.
a-dub 2y ago
a good computer system would allow the application programmer to write their code in a synchronous and single threaded manner and all performance optimizations with respect to multiprocessing (and associated resource sharing) would be handled automatically.
- cempaka 2y ago
  I'm not sure providing that level of complex logic and scheduling is really a proper task for OS kernels. It's essentially what you get from green threads / coroutines which are provided by numerous runtimes and frameworks, but those have to be translated to the various different async IO facilities in order to be portable between OSes.
  - klabb3 2y ago
    Disagree. Scheduling is exactly the type of thing the OS should be doing. It’s (a) extremely easy to get wrong and cause starvation and other problems (b) very complex and large in general and most importantly (c) the OS is the only component with birds-eye view across all processes and hardware (like Numa cores). Every major OS is already doing it, but not for async IO. So it’s absolutely reasonable to maintain this responsibility during the transition to async IO. This goes back to my initial point. We should not need a team of PhDs and kernel hackers to do hello world. The old sync syscalls have largely uniformity across platforms, but the async ones don’t. This means, in practice, that most applications, runtimes and thus users, don’t get to enjoy the benefits.
    - vacuity 2y ago
      Counterpoint: the OS should indeed do its job of scheduling, but applications sometimes know better (the ones specifically implemented by programmers who know better). The OS and application must cooperate to do big jobs like scheduling. See Linux's clunky NUMA lack-of-awareness and transparent huge pages blunder for examples of how sole OS control is lacking. This is not fundamentally a Linux implementation issue: the OS does not always know best. The other extreme, giving applications control, is obviously wrong too, like cooperative scheduling being infeasible for general purpose systems. The OS should have strong defaults but also empower applications (or rather, their programmers) to try their utmost if they want. The OS is not the end all, be all. It is the platform on top of which applications gain valuable power, and constraints. That is basically by definition.
badgersnake 2y ago
If you come from the starting point that Linux is correct and everything different is wrong then you get blog posts like this.
- throw16180339 2y ago
  The author thinks that musl, Alpine, and Linux are right and everything else is wrong. * https://ariadne.space/2022/03/27/the-tragedy-of-gethostbyname/ https://ariadne.space/2022/03/27/the-tragedy-of-gethostbynam... - musl's DNS resolver doesn't support TCP lookups, but it's your fault for expecting the standard DNS lookup functions to work. You should change your program to use a third-party library for DNS lookups. This post is a couple years old; musl now supports TCP lookups. * https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/ https://ariadne.space/2021/06/25/understanding-thread-stack-... - Alpine has a stack size that's substantially smaller (128k) than other widely used OSes (512k+). Your program may work everywhere else, but you're wrong to assume a reasonable stack size. Here's how to jump through a hoop and make it work on Alpine.
  - blueflow 2y ago
    Same person that both contributed the code of conduct to Alpine and later got caught bragging on twitter about having bullied people out of the project. The work of someone who did not ask themselves the necessary questions, but now its some years later and things have changed.
  - johnbellone 2y ago
    Sounds like an asshole.
  - a_t48 2y ago
    The second article is not saying that Alpine is right, only that it behaves differently. "Reasonable" stack size is pretty subjective - many workflows won't care about stack allocating megabytes at a time and could save RAM from having a smaller stack size. The article is pretty informative with workarounds for this situation. There's no need to attack the author about this, especially from a throwaway.
WhyNotHugo 2y ago
The only thing that's agreed upon and settled across OSs, is that reading from a regular file is always blocking, even with O_NONBLOCK. This means that you neither epoll nor kqueue nor poll work if you ever need to read from a file. I wish the BSDs and Linux and other platforms would align on non-blocking IO semantics. I just hope it's something new which can handle non-blocking IO with files too.
- p_ing 2y ago
  This is not the case on NT. The OVERLAPPED data structure allows for async file operations. https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o https://learn.microsoft.com/en-us/windows/win32/fileio/synch...

by klabb3 2y ago

BobbyTables2 2y ago

I fully agree. Synchronous I/O looks appealing for beginners and simple projects. It seems like the simplest path for things where I/O isn’t the primary focus. Then one suddenly finds themselves juggling ephemeral threads to “background” tasks or such. The application quickly becomes a multi-threaded monster and still suffers responsiveness issues from synchronous use. The Tower of Babel created from third party runtimes further adds to the mess even for those who try to do better.

worik 2y ago
> Synchronous I/O looks appealing for beginners and simple projects. Yes. And. Synchronous I/O is suitable for almost all projects When it is unsuitable it is catastrophicly unsuitable,
- astrange 2y ago
  Synchronous I/O is actually better than async for many programs, partly because almost all async systems were designed without thinking about priority inversions and can't resolve them properly. If you have futures, you have this problem.
  - astrange 2y ago
    Forgot to say, the correctly designed ones are called "structured concurrency".
  - p_ing 2y ago
    Since the NT kernel is async throughout, it solves this with periodic priority boosts for non-running but runnable threads. I swear, the kernel designers just need to look at NT :-P But we're well beyond a complete rewrite of old kernels, sadly.
    - astrange 2y ago
      That can work but I think it's cheating. Especially if your scheduling is tied into power management or you have high/low power cores, then boosting the low priority threads will waste power.
liontwist 2y ago
The entire unix operating system is designed to abstract away async problems and turn them sync so you can write sane code (if, else, for). In my experience simple blocking code is incredibly misunderstood. Most programmers stop their thinking at “blocking is slow and non blocking is fast!”
- mschuster91 2y ago
  > The entire unix operating system is designed to abstract away async problems and turn them sync so you can write sane code (if, else, for). That only works for workloads that can be independently parallelized. For stuff that cannot be - which is the majority of consumer workload - you're in for a world of pain with locking and whatnot other issues.
  - liontwist 2y ago
    No. It’s almost all the time that your computer has more than one thing to do; - your browser uses a process per tab - your web server responds to multiple requests in parallel - your web app is going to launch other programs - your desktop ui is going to have a compositing server separate from each application - your compiler is going to work on multiple source files
- mzimbres 2y ago
  > In my experience simple blocking code is incredibly misunderstood. I don't think so. You can't even implement a full-duplex protocol properly with sync IO since reading and writing can't happen at the same time on the same thread. Splitting them to different threads also won't help since accessing shared state requires synchronization with e.g. mutex that kills again simultaneous reading and writing.
  - liontwist 2y ago
    How do you propose to implement duplex IO? And what’s the example case? With threads the reading and writing can happen simultaneously. The synchronization is only on whatever data you need to share. Note that if reading and writing are dependent, then single thread actually makes sense because you can’t do one without the other. Otherwise the pattern of logic you will execute for reading or writing is completely different, which is why threading is the perfect abstraction.

jandrewrogers 2y ago

The differences in design reflect differences in use case assumptions, which in turn reflects that different OS tend to be used for different workloads. Broadly speaking it is difficult to use async I/O well without understanding the implementation details. It is a sharp tool for experts, not magic “go faster” juice.

vacuity 2y ago
Which is why parent is calling for standardization, so that people don't usually need to be experts. I reject that there are a diverse range of async abstractions that are all justified such that a given OS only supports some well. More likely, that means the designs are poorly done. Although I can't say anything if it's for backwards compatibility, how I loathe and crave it.
- jandrewrogers 2y ago
  The bigger issue is that no OS was designed for expansive and flexible async functionality. The implementations reflect practical limitations of remaining compatible with the rest of the guarantees the kernel makes. I would love a portable standard, I’ve been doing async for a long time and I would benefit greatly, but I also understand why it hasn’t happened. A good standard would likely require every OS to materially redesign its internals in a way that would have backward compatibility issues for the other users of the system. I am not against the idea since I would clearly benefit personally, but I understand why the tradeoffs make it unlikely to happen. Even going from io_submit to io_uring just on Linux changes the way you need to architect your software, aside from API changes, and that is the same kernel. It deeply touches the behavior of the system.
  - p_ing 2y ago
    > The bigger issue is that no OS was designed for expansive and flexible async functionality. The NT kernel was designed in this very way. All I/O in the NT kernel is async. It was one of the design principles.

bluetomcat 2y ago

The claim is silly and unfounded. If anything, kqueue is the interface that pollutes userspace less. You have a single kevent call that is used both for waiting and registering events represented by (filter, ident) tuples. All of the data related to the event is contained in a struct that’s passed between the kernel and the user. For non-fd events (EVFILT_PROC, EVFILT_TIMER, EVFILT_SIGNAL), it is much more straightforward to use compared to the Linux way where you need to keep track of one more resource, read specific binary data from it, have specific flags, etc.

somat 2y ago
The article reads more like pr damage control than an actual complaint. epoll has a reputation as a fragile, hard to correctly use interface. and kqueue as relatively simple and sane. So the question always is. why didn't linux just adopt the already existing kqueue instead? Sometimes it seems that linux does end up with more than it's fair share of technical excellence coupled to a bad interface. iptables, epoll and git come to mind.
- adrian_b 2y ago
  This other article has been discussed in the past on HN: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-broken-22/ https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-... Because it discusses epoll in much more detail, it is far more convincing than the parent article of this thread. The conclusion of that article is that how to use correctly epoll is not at all obvious and it has some pitfalls that are not easy to avoid. Therefore epoll seems to be more affected by a serious technical debt, i.e. by a less than good design of the original API.

Aurornis 2y ago

> While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? I came to the comments hoping to get more explanation. I was waiting for the full explanation of the technical debt but then the article just came to an abrupt end. I wonder if the article started as a short history and explanation of differences, but then the dramatic headline was chosen to drive more clicks? The article could have been interesting by itself without setting up the “mountain of technical debt” claim in the headline.

mst 2y ago
I know ariadne well enough that I'm pretty confident that the headline will have been chosen out of annoyance rather than click maximisation. That doesn't mean I agree with the conclusion (I am ambivalent and would have to think rather more before having an opinion myself) but I'm reasonably sure of the motivation nonetheless.

a-dub 2y ago

a good computer system would allow the application programmer to write their code in a synchronous and single threaded manner and all performance optimizations with respect to multiprocessing (and associated resource sharing) would be handled automatically.

cempaka 2y ago
I'm not sure providing that level of complex logic and scheduling is really a proper task for OS kernels. It's essentially what you get from green threads / coroutines which are provided by numerous runtimes and frameworks, but those have to be translated to the various different async IO facilities in order to be portable between OSes.
- klabb3 2y ago
  Disagree. Scheduling is exactly the type of thing the OS should be doing. It’s (a) extremely easy to get wrong and cause starvation and other problems (b) very complex and large in general and most importantly (c) the OS is the only component with birds-eye view across all processes and hardware (like Numa cores). Every major OS is already doing it, but not for async IO. So it’s absolutely reasonable to maintain this responsibility during the transition to async IO. This goes back to my initial point. We should not need a team of PhDs and kernel hackers to do hello world. The old sync syscalls have largely uniformity across platforms, but the async ones don’t. This means, in practice, that most applications, runtimes and thus users, don’t get to enjoy the benefits.
  - vacuity 2y ago
    Counterpoint: the OS should indeed do its job of scheduling, but applications sometimes know better (the ones specifically implemented by programmers who know better). The OS and application must cooperate to do big jobs like scheduling. See Linux's clunky NUMA lack-of-awareness and transparent huge pages blunder for examples of how sole OS control is lacking. This is not fundamentally a Linux implementation issue: the OS does not always know best. The other extreme, giving applications control, is obviously wrong too, like cooperative scheduling being infeasible for general purpose systems. The OS should have strong defaults but also empower applications (or rather, their programmers) to try their utmost if they want. The OS is not the end all, be all. It is the platform on top of which applications gain valuable power, and constraints. That is basically by definition.

badgersnake 2y ago

If you come from the starting point that Linux is correct and everything different is wrong then you get blog posts like this.

throw16180339 2y ago
The author thinks that musl, Alpine, and Linux are right and everything else is wrong. * https://ariadne.space/2022/03/27/the-tragedy-of-gethostbyname/ https://ariadne.space/2022/03/27/the-tragedy-of-gethostbynam... - musl's DNS resolver doesn't support TCP lookups, but it's your fault for expecting the standard DNS lookup functions to work. You should change your program to use a third-party library for DNS lookups. This post is a couple years old; musl now supports TCP lookups. * https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/ https://ariadne.space/2021/06/25/understanding-thread-stack-... - Alpine has a stack size that's substantially smaller (128k) than other widely used OSes (512k+). Your program may work everywhere else, but you're wrong to assume a reasonable stack size. Here's how to jump through a hoop and make it work on Alpine.
- blueflow 2y ago
  Same person that both contributed the code of conduct to Alpine and later got caught bragging on twitter about having bullied people out of the project. The work of someone who did not ask themselves the necessary questions, but now its some years later and things have changed.
- johnbellone 2y ago
  Sounds like an asshole.
- a_t48 2y ago
  The second article is not saying that Alpine is right, only that it behaves differently. "Reasonable" stack size is pretty subjective - many workflows won't care about stack allocating megabytes at a time and could save RAM from having a smaller stack size. The article is pretty informative with workarounds for this situation. There's no need to attack the author about this, especially from a throwaway.

WhyNotHugo 2y ago

The only thing that's agreed upon and settled across OSs, is that reading from a regular file is always blocking, even with O_NONBLOCK. This means that you neither epoll nor kqueue nor poll work if you ever need to read from a file. I wish the BSDs and Linux and other platforms would align on non-blocking IO semantics. I just hope it's something new which can handle non-blocking IO with files too.

p_ing 2y ago
This is not the case on NT. The OVERLAPPED data structure allows for async file operations. https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o https://learn.microsoft.com/en-us/windows/win32/fileio/synch...