4 ms·

I am not at up to date on this either, but as far as I know, the old status quo was this: For non-blocking sockets and the like you have what grew out of the r

I am not at up to date on this either, but as far as I know, the old status quo was this: For non-blocking sockets and the like you have what grew out of the readiness model from select and poll. Today on Linux that's epoll. This model never worked for files on disk. There is the POSIX aio API for that. On Linux, glibc implements that and it's always been based on synchronous I/O in thread pools because the kernel support isn't good. And I guess there's been Linux specific things like io_submit(2), which as the article states only worked with O_DIRECT and sometimes fell back to synchronous I/O. It says the new mechanism works with both files and sockets, but I imagine it's still reasonable to use epoll for sockets, and only use this new stuff for files on disk. One good thing about the readiness model for sockets is you don't need to allocate memory upfront every potential read the remote host might trigger.

Klathmon 8y ago
This explanation gave me that feeling of how utterly incredible it is at any software works as well as it does. The amount of work that goes into supporting, debugging, developing, improving, and designing the sheer number of ways that we do things, at literally hundreds of layers of abstractions all which are slightly wrong and each of which does things their own way with their own tradeoffs. I'm damn glad there are people a lot smarter and more dedicated than I am that keep this stuff running so that I can store a value in localstorage in a browser and have it punch through all of those layers in a fraction of a moment down to writing bits on one of like a dozen storage mediums on any number of OSs via multiple interfaces in multiple browsers.
aarongolliver 8y ago
Can you (or someone) explain _why_ the model translate to files on disk? Is it that there's a different set of operations that need to be supported? (you can't seek a socket I suppose)
- asveikau 8y ago
  The model for aio is "here is a buffer, let me know when you've done the operation". Your request can sit in a queue somewhere or whatever for as long as it takes. The model for the various FD polling mechanisms is "wake me up when read(2) won't return EAGAIN when I call it later". That is very natural for sockets, because what read(2) is doing is draining a kernel side receive buffer, so the question is "wake me up when the buffer isn't empty". If the FD were a file on disk, knowing ahead of time that read(2) isn't going to return quickly is kind of a clumsy task.
  - bogomipz 8y ago
    >"If the FD were a file on disk, knowing ahead of time that read(2) isn't going to return quickly is kind of a clumsy task." But isn't this exactly what happens? A process running in kernel mode calls schedule() and sleeps waiting for I/O. The reason for calling schedule() is that it already does know read isn't going to return quickly. But maybe I'm completely misunderstanding your explanation about why current AIO on Linux is clunky?
    - asveikau 8y ago
      The key observation is this ahead of time thing. If you've reached the point of invoking the scheduler you have already done a bunch of work to decide that, based on a request that has already started. On the other hand if epoll_wait returned and said the fd was ready, but the read hasn't yet been issued, who is to say the conditions will be the same later? Keep in mind also that other processes can do things that evicts caches, or make other requests to the same disk or controller. For all of these cases I suppose you could make the I/O fail with EAGAIN, but this seems like a lot of potential for spurious wakeups.
      - bogomipz 8y ago
        Thanks for the clarification, that makes good sense. Cheers.

Klathmon 8y ago

This explanation gave me that feeling of how utterly incredible it is at any software works as well as it does. The amount of work that goes into supporting, debugging, developing, improving, and designing the sheer number of ways that we do things, at literally hundreds of layers of abstractions all which are slightly wrong and each of which does things their own way with their own tradeoffs. I'm damn glad there are people a lot smarter and more dedicated than I am that keep this stuff running so that I can store a value in localstorage in a browser and have it punch through all of those layers in a fraction of a moment down to writing bits on one of like a dozen storage mediums on any number of OSs via multiple interfaces in multiple browsers.

aarongolliver 8y ago

Can you (or someone) explain _why_ the model translate to files on disk? Is it that there's a different set of operations that need to be supported? (you can't seek a socket I suppose)

asveikau 8y ago
The model for aio is "here is a buffer, let me know when you've done the operation". Your request can sit in a queue somewhere or whatever for as long as it takes. The model for the various FD polling mechanisms is "wake me up when read(2) won't return EAGAIN when I call it later". That is very natural for sockets, because what read(2) is doing is draining a kernel side receive buffer, so the question is "wake me up when the buffer isn't empty". If the FD were a file on disk, knowing ahead of time that read(2) isn't going to return quickly is kind of a clumsy task.
- bogomipz 8y ago
  >"If the FD were a file on disk, knowing ahead of time that read(2) isn't going to return quickly is kind of a clumsy task." But isn't this exactly what happens? A process running in kernel mode calls schedule() and sleeps waiting for I/O. The reason for calling schedule() is that it already does know read isn't going to return quickly. But maybe I'm completely misunderstanding your explanation about why current AIO on Linux is clunky?
  - asveikau 8y ago
    The key observation is this ahead of time thing. If you've reached the point of invoking the scheduler you have already done a bunch of work to decide that, based on a request that has already started. On the other hand if epoll_wait returned and said the fd was ready, but the read hasn't yet been issued, who is to say the conditions will be the same later? Keep in mind also that other processes can do things that evicts caches, or make other requests to the same disk or controller. For all of these cases I suppose you could make the I/O fail with EAGAIN, but this seems like a lot of potential for spurious wakeups.
    - bogomipz 8y ago
      Thanks for the clarification, that makes good sense. Cheers.