8 ms·

POSIX AIO is not non-blocking async I/O; it can block other threads requesting the resource. IOCP is a true non-blocking async I/O. IOCP also extends to all for

POSIX AIO is not non-blocking async I/O; it can block other threads requesting the resource. IOCP is a true non-blocking async I/O. IOCP also extends to all forms of I/O (file, TCP socket, network, mail slot, pipes, etc.) instead of a particular type. POSIX AIO has usability problems also outlined in the previously linked thread. Remember, all I/O in NT is async at the kernel level. It's not a "bolt-on". IoRing is limited to file reads, unlike io_uring.

mananaysiempre 2y ago
POSIX AIO + FreeBSD kqueue or Solaris ports are functionally equivalent to IOCP as far as I can tell.
- nullindividual 2y ago
  This describes kqueue: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores?slide=37 https://speakerdeck.com/trent/pyparallel-how-we-removed-the-...
  - trentnelson 2y ago
    I should do an updated version of that deck with io_uring and sans the PyParallel element. I still think it’s a good resource for depicting the differences in I/O between NT & UNIX. And yeah, IOCP has implicit awareness of concurrency, and can schedule optimal threads to service a port automatically. There hasn’t been a way to do that on UNIX until io_uring.
    - nullindividual 2y ago
      Yes, please! And if you're interested, RegisteredIO and I assume you'd drop in IoRing. In a nicely wrapped PDF :-)
      - trentnelson 2y ago
        Yeah I’d definitely include RegisteredIO and IoRing. When I was interviewing at Microsoft a few years back, I was actually interviewed by the chap that wrote RegisteredIO! Thought that was neat.
        deleted 2y ago
        [deleted]
deleted 2y ago
[deleted]
netbsdusers 2y ago
Posix AIO is just an interface. Windows also relies on thread pools for some async io (I.e. when reading files when all the data necessary to generate a disk request isn't in cache - good luck writing that as purely asynchronous)
jhallenworld 2y ago
This is all fine, but Window-NT file access is still slow compared with Linux- this shows up in shell scripts. The reason is supposedly that it insists on syncing during close, or maybe waiting for all closed files to sync before allowing the process to terminate. Shouldn't close finality be an optional async event or something?
- nullindividual 2y ago
  The reason is due to file system filters, of which Windows Defender is always there. There is a significant delay from Defender when performing CloseFile()[0]. > As I was looking at the raw system calls related to I/O, something immediately popped out: CloseFile() operations were frequently taking 1-5 milliseconds whereas other operations like opening, reading, and writing files only took 1-5 microseconds. That's a 1000x difference! This is why DevDrive was introduced[1]. You can either have Defender operate in async mode (default) or remove it entirely from the volume at your own risk. The performance issue isn't related to sync or async I/O. [0] https://gregoryszorc.com/blog/2015/10/22/append-i/o-performance-on-windows/ https://gregoryszorc.com/blog/2015/10/22/append-i/o-performa... [1] https://devblogs.microsoft.com/visualstudio/devdrive/ https://devblogs.microsoft.com/visualstudio/devdrive/
  - cyberax 2y ago
    Windows FS stack is still _way_ slower than Linux. Filesystem operations have to create IRPs and submit them for execution through a generic mechanism. These IRPs can get filtered and modified in-flight, providing quite a bit of overall flexibility. In Linux, filesystem paths are super-optimized, with all the filtering (e.g. for SELinux) special-cased if needed. But even still, Windows also had to cheat to avoid completely cratering the performance, there's a shortcut called "FastIO": https://learn.microsoft.com/en-us/windows-hardware/drivers/ifs/registering-fast-i-o-dispatch-routines https://learn.microsoft.com/en-us/windows-hardware/drivers/i... I wrote a filesystem for Windows around 25 years ago, and I still remember how I implemented all the required prototypes and everything in Explorer worked. But notepad.exe was just showing me empty data. It took me several days to find a note tucked into MSDN that you need to implement FastIO for memory mapped files to work (which Notepad.exe used).
Sesse__ 2y ago
> Remember, all I/O in NT is async at the kernel level. It's not a "bolt-on". All I/O in Linux is also async at the kernel level! The problem has always been expressing that asynchronicity to userspace in a sane way.
- netbsdusers 2y ago
  Filesystem io (and probably more) is not async at the kernel level in Linux. (Just imagine trying to express the complexity of it in continuations or some sort od state machine!) As such io_uring takes the form of a kernel thread pool. Disk block io by contrast is much easier to be fundamentally async since its almost always a case of submitting a request to an HBA and waiting for an interrupt.
  - wmf 2y ago
    Just imagine trying to express the complexity of [a filesystem] in continuations or some sort of state machine! Arguably asyc/await could help with this; obviously it didn't exist in 1991 when Linux was created but it would be interesting to revisit this topic.
    - SoothingSorbet 2y ago
      > Arguably asyc/await could help with this; obviously it didn't exist in 1991 when Linux was created Wouldn't that just consist of I/O operations returning futures and then having an await() block the calling thread until the future is done (i.e. put it on a waitqueue)?
  - treyd 2y ago
    > Just imagine trying to express the complexity of it in continuations or some sort od state machine! With Rust in the kernel this becomes somewhat possible to conceptualize.
  - mananaysiempre 2y ago
    > Just imagine trying to express the complexity of it in continuations or some sort o[f] state machine! You’d probably want to use either some sort of code generation to do the requisite CPS transform[1] or the Duff’s-device-like preprocessor trick[2], but it’s definitely doable with some compiler support. Not in an existing codebase, though. (Brought to you by working on a C codebase that does express stuff like this as explicit callbacks and context structures. Ugh.) [1] https://dx.doi.org/10.1007/s10990-012-9084-5 https://dx.doi.org/10.1007/s10990-012-9084-5, https://www.irif.fr/~jch/research/cpc-2012.pdf https://www.irif.fr/~jch/research/cpc-2012.pdf, https://github.com/kerneis/cpc https://github.com/kerneis/cpc [2] https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html
dataflow 2y ago
> IOCP is a true non-blocking async I/O. Unfortunately that's only half-true. You can (and will) still get blockage sometimes with IOCP, it depends on a lot of factors, like how loaded the system is, I think. There is absolutely no guarantee that your I/O will actually occur asynchronously, only that you will be notified of its completion asynchronously. Also, opening a file is also always synchronous, which is quite annoying if you're trying not to block e.g. a UI thread. The implication of both of these is you still need dedicated I/O threads. I love IOCP as much as anyone, but it does have these flaws, and was very much designed to be used from multiple threads. The only workaround I'm aware of was User-Mode Scheduling, which effectively notified you as soon as your thread got blocked, but it still required multiple logical threads, and Microsoft removed support for it in Windows 11.

mananaysiempre 2y ago

POSIX AIO + FreeBSD kqueue or Solaris ports are functionally equivalent to IOCP as far as I can tell.

nullindividual 2y ago
This describes kqueue: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores?slide=37 https://speakerdeck.com/trent/pyparallel-how-we-removed-the-...
- trentnelson 2y ago
  I should do an updated version of that deck with io_uring and sans the PyParallel element. I still think it’s a good resource for depicting the differences in I/O between NT & UNIX. And yeah, IOCP has implicit awareness of concurrency, and can schedule optimal threads to service a port automatically. There hasn’t been a way to do that on UNIX until io_uring.
  - nullindividual 2y ago
    Yes, please! And if you're interested, RegisteredIO and I assume you'd drop in IoRing. In a nicely wrapped PDF :-)
    - trentnelson 2y ago
      Yeah I’d definitely include RegisteredIO and IoRing. When I was interviewing at Microsoft a few years back, I was actually interviewed by the chap that wrote RegisteredIO! Thought that was neat.
      - deleted 2y ago
        [deleted]

deleted 2y ago

[deleted]

netbsdusers 2y ago

Posix AIO is just an interface. Windows also relies on thread pools for some async io (I.e. when reading files when all the data necessary to generate a disk request isn't in cache - good luck writing that as purely asynchronous)

jhallenworld 2y ago

This is all fine, but Window-NT file access is still slow compared with Linux- this shows up in shell scripts. The reason is supposedly that it insists on syncing during close, or maybe waiting for all closed files to sync before allowing the process to terminate. Shouldn't close finality be an optional async event or something?

nullindividual 2y ago
The reason is due to file system filters, of which Windows Defender is always there. There is a significant delay from Defender when performing CloseFile()[0]. > As I was looking at the raw system calls related to I/O, something immediately popped out: CloseFile() operations were frequently taking 1-5 milliseconds whereas other operations like opening, reading, and writing files only took 1-5 microseconds. That's a 1000x difference! This is why DevDrive was introduced[1]. You can either have Defender operate in async mode (default) or remove it entirely from the volume at your own risk. The performance issue isn't related to sync or async I/O. [0] https://gregoryszorc.com/blog/2015/10/22/append-i/o-performance-on-windows/ https://gregoryszorc.com/blog/2015/10/22/append-i/o-performa... [1] https://devblogs.microsoft.com/visualstudio/devdrive/ https://devblogs.microsoft.com/visualstudio/devdrive/
- cyberax 2y ago
  Windows FS stack is still _way_ slower than Linux. Filesystem operations have to create IRPs and submit them for execution through a generic mechanism. These IRPs can get filtered and modified in-flight, providing quite a bit of overall flexibility. In Linux, filesystem paths are super-optimized, with all the filtering (e.g. for SELinux) special-cased if needed. But even still, Windows also had to cheat to avoid completely cratering the performance, there's a shortcut called "FastIO": https://learn.microsoft.com/en-us/windows-hardware/drivers/ifs/registering-fast-i-o-dispatch-routines https://learn.microsoft.com/en-us/windows-hardware/drivers/i... I wrote a filesystem for Windows around 25 years ago, and I still remember how I implemented all the required prototypes and everything in Explorer worked. But notepad.exe was just showing me empty data. It took me several days to find a note tucked into MSDN that you need to implement FastIO for memory mapped files to work (which Notepad.exe used).

Sesse__ 2y ago

> Remember, all I/O in NT is async at the kernel level. It's not a "bolt-on". All I/O in Linux is also async at the kernel level! The problem has always been expressing that asynchronicity to userspace in a sane way.

netbsdusers 2y ago
Filesystem io (and probably more) is not async at the kernel level in Linux. (Just imagine trying to express the complexity of it in continuations or some sort od state machine!) As such io_uring takes the form of a kernel thread pool. Disk block io by contrast is much easier to be fundamentally async since its almost always a case of submitting a request to an HBA and waiting for an interrupt.
- wmf 2y ago
  Just imagine trying to express the complexity of [a filesystem] in continuations or some sort of state machine! Arguably asyc/await could help with this; obviously it didn't exist in 1991 when Linux was created but it would be interesting to revisit this topic.
  - SoothingSorbet 2y ago
    > Arguably asyc/await could help with this; obviously it didn't exist in 1991 when Linux was created Wouldn't that just consist of I/O operations returning futures and then having an await() block the calling thread until the future is done (i.e. put it on a waitqueue)?
- treyd 2y ago
  > Just imagine trying to express the complexity of it in continuations or some sort od state machine! With Rust in the kernel this becomes somewhat possible to conceptualize.
- mananaysiempre 2y ago
  > Just imagine trying to express the complexity of it in continuations or some sort o[f] state machine! You’d probably want to use either some sort of code generation to do the requisite CPS transform[1] or the Duff’s-device-like preprocessor trick[2], but it’s definitely doable with some compiler support. Not in an existing codebase, though. (Brought to you by working on a C codebase that does express stuff like this as explicit callbacks and context structures. Ugh.) [1] https://dx.doi.org/10.1007/s10990-012-9084-5 https://dx.doi.org/10.1007/s10990-012-9084-5, https://www.irif.fr/~jch/research/cpc-2012.pdf https://www.irif.fr/~jch/research/cpc-2012.pdf, https://github.com/kerneis/cpc https://github.com/kerneis/cpc [2] https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html https://www.chiark.greenend.org.uk/~sgtatham/coroutines.html

dataflow 2y ago

> IOCP is a true non-blocking async I/O. Unfortunately that's only half-true. You can (and will) still get blockage sometimes with IOCP, it depends on a lot of factors, like how loaded the system is, I think. There is absolutely no guarantee that your I/O will actually occur asynchronously, only that you will be notified of its completion asynchronously. Also, opening a file is also always synchronous, which is quite annoying if you're trying not to block e.g. a UI thread. The implication of both of these is you still need dedicated I/O threads. I love IOCP as much as anyone, but it does have these flaws, and was very much designed to be used from multiple threads. The only workaround I'm aware of was User-Mode Scheduling, which effectively notified you as soon as your thread got blocked, but it still required multiple logical threads, and Microsoft removed support for it in Windows 11.