8 ms·

I'm surprised there is no mention of the lack of non-blocking filesystem reads and opens. In other words, I should be able to submit a read request, and then s

I'm surprised there is no mention of the lack of non-blocking filesystem reads and opens. In other words, I should be able to submit a read request, and then select()/poll()/epoll() for when the data is available like network I/O.

geocar 9y ago
Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() so this doesn't actually help read from disks (even when they're backed by network/shared storage). That's why you often see this kind of seeming "nonsense" in HPC startup scripts: for x in *; do cat "$x" & done | pipeline ... Furthermore, in a networked-environment, select()/poll()/epoll() actually decrease concurrency compared to threads because now you need another system call to go fetch the data from the kernel. If I've got the CPUs I'd rather use threads. In the Windows model (in contrast), the programmer says "when this data is available (from this socket, file, etc), put it here (into this memory buffer)" and then waits for the kernel to do something. As a result, you skip that extra syscall -- but you also skip any other read you might have needed to do for data that's arrived at (roughly) the same time. AIO[1] is potentially similar, but it never seems to benchmark well in my experience. [1]: http://man7.org/linux/man-pages/man7/aio.7.html http://man7.org/linux/man-pages/man7/aio.7.html
- FooBarWidget 9y ago
  What is surprising to me is that it is 2017 and nobody has attempted to solve these problems. Maybe the current mechanisms, with all their faults, are considered good enough?
  - AstralStorm 9y ago
    Generally epoll and event io are good enough for millions of simultaneous files or sockets. Nobody tells you to use direct read or write instead of mmap either.
    - cturner 9y ago
      Mmap - I can't think of a scenario where it improves over read or write in the context discussed here: async access. Mmap is synchronous, and you can't do select/poll on it.
      - geocar 9y ago
        https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt
        cturner 9y ago
        Not sure why you were downvoted. That's some seriously hairy stuff there. I reckon I'd move to building my app as a kernel module more readily than trying to make a robust async arrangement using that mechanism :) Call me demanding, but I think we should can reasonably expect easier access to async from our OS than this mechanism offers. On that topic of which.. how's the future of operating systems coming along?
  - dom0 9y ago
    Other systems (Windows and I think Solaris, but some more obscure systems likely do this as well) can do asynchronous disk IO, but it's not perfect, either. I think the main reason Linux doesn't really do it is that asynchronous IO is an intrusive change and there would be just too much work to implement it in all the file systems etc.; i.e. "not worth it". Many applications which are not OK with synchronous disk I/O seem to find thread pools good enough: reasonably easy to implement, reasonably portable, usually performs ok.
    - adrianratnapala 9y ago
      I only half buy the "not worth it". Traditional unix, including Linux pretends to support poll() etc for file I/O (i.e. it supports the interfaces, but the reality is synchronous). The reason for this is that an application that expects async I/O can kinda-sorta-work with real files -- and "kinda sorta" will not be too bad because local disks are fast enough to paper over any problem. But then if the disks aren't local, why not actually make non-blocking I/O work as advertised? Actually I can guess the answere: there are 101 corner cases that mean I can't neatly separate out the apps design for noblocking I/O. And that's why I half buy the argument. But only half.
  - mfukar 9y ago
    Those problems are already solved by async I/O in platforms other than Linux.
  - geocar 9y ago
    I think that's unfair: A lot of people have attempted to solve these problems(), and the Windows Way™ has it's own problems. The real answer is probably between elastic infrastructure and the fact that computers got faster and cheaper, and so -- at least for a while the number of people needing solutions here went down. https://lwn.net/Articles/219954/ https://lwn.net/Articles/219954/ * https://lwn.net/Articles/223899/ https://lwn.net/Articles/223899/ * https://lwn.net/Articles/221913/ https://lwn.net/Articles/221913/ * https://lwn.net/Articles/316806/ https://lwn.net/Articles/316806/ * https://lwn.net/Articles/612038/ https://lwn.net/Articles/612038/ * https://lwn.net/Articles/671658/ https://lwn.net/Articles/671658/ * http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt * http://ogris.de/howtos/splice.html http://ogris.de/howtos/splice.html And so on. However a few companies have seen real commercial benefit from actually analysing their data, so we're seeing a lot of "big data" interest. Some of these companies have noticed that a bunch of hadoop/mongo/whatever boxes can't even approach the performance of one "big" box tuned right, so we're seeing somewhat of a resurgence in interest here.
    - int_19h 9y ago
      > and the Windows Way™ has it's own problems Could you expand on that? (I'm assuming that by the "Windows Way", you mean I/O completion ports.)
- gpderetta 9y ago
  > Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() There is no other choice really. Let's say you want to read a specific page from a file. No page is currently in the kernel buffers. If poll where to return non-ready (and remember that poll has no idea which pages you are interested in), what action, within posix, would you take to change it? poll and friends just do not make sense for plain file descriptors. In linux, splice add an interesting twist, but it is still awkward to use.
  - geocar 9y ago
    > Let's say you want to read a specific page from a file. pread() is an unusual case. I'd be happy if poll() simply indicated read() would succeed, and I'm happy doing an lseek() before every read(). However a pread() with O_NONBLOCK could return EAGAIN and poll() would then know what pages I'm interested in. > what action, within posix, would you take to change it? The general (POSIX-compatible) strategy (as I indicated) has been to fork a thread off to handle reading so that my application is notified when the data is in the fifo. This works for me when I'm reading (say) five files from a slow (or remote) storage, but it's not ideal since it causes a copy.
    - gpderetta 9y ago
      poll can't guarantee that the next read will succeed because first of all doesn't know how many pages you are interested in reading and any relevant page in the page cache might be evicted between the poll and the read. Also having pread, which is designed for concurrency and scalability, carry state around for poll (an unbounded amount of state btw: you can have multiple threads issuing concurrent preads on the same fd) makes it completely pointless. There are linux patches floating to add flags to preadv2 so that on a page not present it returns EAGAIN, so that you can optimistically do a sync read an fall back to a thread pool in the slow case. It still doesn't involve poll and friend.
      - geocar 9y ago
        poll() can't guarantee that the next read will succeed anyway, because something else might drain the socket or fifo (e.g. another thread). Programmers simply use O_NONBLOCK anyway and use poll() is wait until the process should try again.
- crest 9y ago
  In theory FreeBSD supports a sane aio_read/write/fsync() implementation by extending sigevent to include the generation of kqueue events, but there are no aio_open/close/fstat/rename() syscalls and if you need a thread pool to deal with those remaining blocking file system calls there are few advantages to using the in-kernel thread pool implementing the aio_*() syscalls. An other problem is that filesystem aio calls are limited to local file systems, because accesses to network file systems could block indefinitely starving local file system accesses by blocking all threads in the pool.
api 9y ago
I thought you could, at least on Linux.
- sgift 9y ago
  You can on (more or less) any modern OS, but it's not part of POSIX, it's platform specific.
  - xfer 9y ago
    It is part of posix, it's called posix AIO, but working with it is a massive pain.
fulafel 9y ago
Threads are less of a problem with file i/o, because usually a process doesn't need to have 10000 outstanding disk io requests. (Windows async disk i/o is thread-backed too behind the scenes, though)
- useerup 9y ago
  Windows async I/O ("overlapped" in Windows parlance) is backed by a thread pool (completion ports). Windows does not spin up or allocate a thread just to wait for an IO operation to complete. Rather, when an IO operation completes, a thread is allocated from the thread pool (hence the name completion ports) on which the callback happens. So there is no excessive allocation of threads. Everything is event-driven and kernel managed until the operation completes.
  - emn13 9y ago
    Simultaneously, people overestimate the costs of threads, and underestimate the efficiencies they provide. At extremely high loads, they become an issue, but it's quite rare. And at normal loads, threading is (IME) often simply more efficient than typical async solutions in practice, probably since a thread gets synchronized access to its internal state and consistent ordering of execution (almost) for free, whereas async code needs to (usually) reimplement that in user mode. People have taken the notion that threads are slow in some circumstances and made the conclusion that threads are slow, period. But that's simply not true.
    - dom0 9y ago
      The rate of computation is obviously very similar for threads and async, but the rate of multiplexing is not. Switching between processes or threads takes more time than switching between fibers/coroutines. So the general observation that for massively multiplexed workloads fibers/coroutines are lower overhead is generally correct, but the more interesting question is of course "what qualifies as a massively multiplexed workload?" ... I'd argue most kinds of web application servers don't.
exDM69 9y ago
You can't do that with normal read/write and select/poll/epoll, but you can either use aio (posix) or io_submit (linux-specific). In my opinion, a big issue with these is that asynchronous i/o requests can not be used with select/poll/epoll, but rely on other notification methods instead. AIO can use unix signals (ugh) or thread notification (see sigevent(7)) to signal completion. io_submit uses io_getevents to read events about completed I/O requests. From a userspace programmer's point of view, it would make much more sense if io_setup() would return a file descriptor that could be used with select/poll, and io_events would be read() from the fd. Currently it's impossible (?) to have filesystem i/o in the same select/poll loop with your network i/o because of this issue. This is why libuv falls back to using threads and synchronous i/o to provide "async" i/o.
- MichaelMoser123 9y ago
  you can do io_submit from one thread and then have some dedicated threads do io_getevent + event signalling (comes at a price though). Still all this event goodness is not good enough for things like infiniband and the brave world of zero copy networking.
  - exDM69 9y ago
    Yeah, sure that kinda works but it won't play nice with having an event loop on network events. A typical high performance server has multiple worker threads waiting on events on a select/epoll and if you add in another thread (or several) waiting on io_getevent you end up in a rather contrived solution. You could use a pipe or eventfd or some other fd-based mechanism to wake up a worker thread on a filesystem event, but then you get two context switches (one to wake up io_getevent thread and another to wake up a worker). This carries a lot of indirect costs from jumping back and forth to the kernel. I am unaware of a solution that would make filesystem i/o and network i/o play nice in one event loop without adding some overhead.
- derefr 9y ago
  > Currently it's impossible (?) to have filesystem i/o in the same select/poll loop with your network i/o because of this issue. Hacky workaround: don't do filesystem i/o in the first place. Have the kernel serve an iSCSI target, then consume it over the loopback interface entirely in userspace. Your disk now has network semantics! (But you have to bring your own filesystem driver.) If you don't mind some extra abstraction overhead, you can skip past iSCSI and instead build a userspace NFS client lib into your app, consuming a loopback NFS share. s/NFS/9P/ if your disk server has in-kernel 9P. (Is there any use to these approaches? Yes: they're actually nearly-optimal ways to persist state from a unikernel running on a hypervisor, because all the disks are remote one way or another anyway. Erlang on Xen uses the 9P approach, for example.)
joosters 9y ago
AFAIK, there's no reason why an OS can't make filesystem I/O non-blocking with the standard read/write/select (or poll or whatever) system calls. The only tricky part (API-side) is that write() is defined to confirm that some data got written, or not. If you let write() become non-blocking, and deferred the I/O, perhaps by returning immediately with EAGAIN or some such response, the write is is an unknown state to the rest of the application. What data can then be expected from other read() calls on that file from the same process? The old data? The new? A mix? You'd have to drop the POSIX consistency guarantee. For most programs, this is never likely to be an issue, as they don't tend to be re-reading bits of a file that they are also writing to, so it's a shame there's no option to let the write() be non-blocking if requested. Issue 2 is failures: If the write, when it eventually happens, fails, you've got to have some way of passing that information back to the process. The solution for this seems easy enough; just let the FD become writeable again and the program will probably re-attempt the same write, at which time you can report the previous error. Finally, making open() calls non-blocking would be the real API changer. If you defer all of the file opening work and give the application a file descriptor before doing any I/O, then you can hit all kinds of errors (permissions problems, file not found, and so on). How do you pass them back to the program? You can make any future read() or write() calls fail but these syscalls would now have to return error codes that they aren't specified as actually being able to return (read can't return things like EEXIST, EACCESS and so on). So the API is inadequate for these situations. You can see how they could be extended to cover these cases though.
- jzwinck 9y ago
  File write could work the same as socket send. If the data cannot definitely be written now, write nothing and allow the application to try again, probably after waiting for readiness via poll or select.
- hoytech 9y ago
  See gpderetta's comment elsewhere in this thread: the page-cache is different than socket mbufs in some pretty significant ways.

geocar 9y ago

Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() so this doesn't actually help read from disks (even when they're backed by network/shared storage). That's why you often see this kind of seeming "nonsense" in HPC startup scripts: for x in *; do cat "$x" & done | pipeline ... Furthermore, in a networked-environment, select()/poll()/epoll() actually decrease concurrency compared to threads because now you need another system call to go fetch the data from the kernel. If I've got the CPUs I'd rather use threads. In the Windows model (in contrast), the programmer says "when this data is available (from this socket, file, etc), put it here (into this memory buffer)" and then waits for the kernel to do something. As a result, you skip that extra syscall -- but you also skip any other read you might have needed to do for data that's arrived at (roughly) the same time. AIO[1] is potentially similar, but it never seems to benchmark well in my experience. [1]: http://man7.org/linux/man-pages/man7/aio.7.html http://man7.org/linux/man-pages/man7/aio.7.html

FooBarWidget 9y ago
What is surprising to me is that it is 2017 and nobody has attempted to solve these problems. Maybe the current mechanisms, with all their faults, are considered good enough?
- AstralStorm 9y ago
  Generally epoll and event io are good enough for millions of simultaneous files or sockets. Nobody tells you to use direct read or write instead of mmap either.
  - cturner 9y ago
    Mmap - I can't think of a scenario where it improves over read or write in the context discussed here: async access. Mmap is synchronous, and you can't do select/poll on it.
    - geocar 9y ago
      https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt
      - cturner 9y ago
        Not sure why you were downvoted. That's some seriously hairy stuff there. I reckon I'd move to building my app as a kernel module more readily than trying to make a robust async arrangement using that mechanism :) Call me demanding, but I think we should can reasonably expect easier access to async from our OS than this mechanism offers. On that topic of which.. how's the future of operating systems coming along?
- dom0 9y ago
  Other systems (Windows and I think Solaris, but some more obscure systems likely do this as well) can do asynchronous disk IO, but it's not perfect, either. I think the main reason Linux doesn't really do it is that asynchronous IO is an intrusive change and there would be just too much work to implement it in all the file systems etc.; i.e. "not worth it". Many applications which are not OK with synchronous disk I/O seem to find thread pools good enough: reasonably easy to implement, reasonably portable, usually performs ok.
  - adrianratnapala 9y ago
    I only half buy the "not worth it". Traditional unix, including Linux pretends to support poll() etc for file I/O (i.e. it supports the interfaces, but the reality is synchronous). The reason for this is that an application that expects async I/O can kinda-sorta-work with real files -- and "kinda sorta" will not be too bad because local disks are fast enough to paper over any problem. But then if the disks aren't local, why not actually make non-blocking I/O work as advertised? Actually I can guess the answere: there are 101 corner cases that mean I can't neatly separate out the apps design for noblocking I/O. And that's why I half buy the argument. But only half.
- mfukar 9y ago
  Those problems are already solved by async I/O in platforms other than Linux.
- geocar 9y ago
  I think that's unfair: A lot of people have attempted to solve these problems(), and the Windows Way™ has it's own problems. The real answer is probably between elastic infrastructure and the fact that computers got faster and cheaper, and so -- at least for a while the number of people needing solutions here went down. https://lwn.net/Articles/219954/ https://lwn.net/Articles/219954/ * https://lwn.net/Articles/223899/ https://lwn.net/Articles/223899/ * https://lwn.net/Articles/221913/ https://lwn.net/Articles/221913/ * https://lwn.net/Articles/316806/ https://lwn.net/Articles/316806/ * https://lwn.net/Articles/612038/ https://lwn.net/Articles/612038/ * https://lwn.net/Articles/671658/ https://lwn.net/Articles/671658/ * http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt * http://ogris.de/howtos/splice.html http://ogris.de/howtos/splice.html And so on. However a few companies have seen real commercial benefit from actually analysing their data, so we're seeing a lot of "big data" interest. Some of these companies have noticed that a bunch of hadoop/mongo/whatever boxes can't even approach the performance of one "big" box tuned right, so we're seeing somewhat of a resurgence in interest here.
  - int_19h 9y ago
    > and the Windows Way™ has it's own problems Could you expand on that? (I'm assuming that by the "Windows Way", you mean I/O completion ports.)
gpderetta 9y ago
> Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() There is no other choice really. Let's say you want to read a specific page from a file. No page is currently in the kernel buffers. If poll where to return non-ready (and remember that poll has no idea which pages you are interested in), what action, within posix, would you take to change it? poll and friends just do not make sense for plain file descriptors. In linux, splice add an interesting twist, but it is still awkward to use.
- geocar 9y ago
  > Let's say you want to read a specific page from a file. pread() is an unusual case. I'd be happy if poll() simply indicated read() would succeed, and I'm happy doing an lseek() before every read(). However a pread() with O_NONBLOCK could return EAGAIN and poll() would then know what pages I'm interested in. > what action, within posix, would you take to change it? The general (POSIX-compatible) strategy (as I indicated) has been to fork a thread off to handle reading so that my application is notified when the data is in the fifo. This works for me when I'm reading (say) five files from a slow (or remote) storage, but it's not ideal since it causes a copy.
  - gpderetta 9y ago
    poll can't guarantee that the next read will succeed because first of all doesn't know how many pages you are interested in reading and any relevant page in the page cache might be evicted between the poll and the read. Also having pread, which is designed for concurrency and scalability, carry state around for poll (an unbounded amount of state btw: you can have multiple threads issuing concurrent preads on the same fd) makes it completely pointless. There are linux patches floating to add flags to preadv2 so that on a page not present it returns EAGAIN, so that you can optimistically do a sync read an fall back to a thread pool in the slow case. It still doesn't involve poll and friend.
    - geocar 9y ago
      poll() can't guarantee that the next read will succeed anyway, because something else might drain the socket or fifo (e.g. another thread). Programmers simply use O_NONBLOCK anyway and use poll() is wait until the process should try again.
crest 9y ago
In theory FreeBSD supports a sane aio_read/write/fsync() implementation by extending sigevent to include the generation of kqueue events, but there are no aio_open/close/fstat/rename() syscalls and if you need a thread pool to deal with those remaining blocking file system calls there are few advantages to using the in-kernel thread pool implementing the aio_*() syscalls. An other problem is that filesystem aio calls are limited to local file systems, because accesses to network file systems could block indefinitely starving local file system accesses by blocking all threads in the pool.

api 9y ago

I thought you could, at least on Linux.

sgift 9y ago
You can on (more or less) any modern OS, but it's not part of POSIX, it's platform specific.
- xfer 9y ago
  It is part of posix, it's called posix AIO, but working with it is a massive pain.

fulafel 9y ago

Threads are less of a problem with file i/o, because usually a process doesn't need to have 10000 outstanding disk io requests. (Windows async disk i/o is thread-backed too behind the scenes, though)

useerup 9y ago
Windows async I/O ("overlapped" in Windows parlance) is backed by a thread pool (completion ports). Windows does not spin up or allocate a thread just to wait for an IO operation to complete. Rather, when an IO operation completes, a thread is allocated from the thread pool (hence the name completion ports) on which the callback happens. So there is no excessive allocation of threads. Everything is event-driven and kernel managed until the operation completes.
- emn13 9y ago
  Simultaneously, people overestimate the costs of threads, and underestimate the efficiencies they provide. At extremely high loads, they become an issue, but it's quite rare. And at normal loads, threading is (IME) often simply more efficient than typical async solutions in practice, probably since a thread gets synchronized access to its internal state and consistent ordering of execution (almost) for free, whereas async code needs to (usually) reimplement that in user mode. People have taken the notion that threads are slow in some circumstances and made the conclusion that threads are slow, period. But that's simply not true.
  - dom0 9y ago
    The rate of computation is obviously very similar for threads and async, but the rate of multiplexing is not. Switching between processes or threads takes more time than switching between fibers/coroutines. So the general observation that for massively multiplexed workloads fibers/coroutines are lower overhead is generally correct, but the more interesting question is of course "what qualifies as a massively multiplexed workload?" ... I'd argue most kinds of web application servers don't.

exDM69 9y ago

You can't do that with normal read/write and select/poll/epoll, but you can either use aio (posix) or io_submit (linux-specific). In my opinion, a big issue with these is that asynchronous i/o requests can not be used with select/poll/epoll, but rely on other notification methods instead. AIO can use unix signals (ugh) or thread notification (see sigevent(7)) to signal completion. io_submit uses io_getevents to read events about completed I/O requests. From a userspace programmer's point of view, it would make much more sense if io_setup() would return a file descriptor that could be used with select/poll, and io_events would be read() from the fd. Currently it's impossible (?) to have filesystem i/o in the same select/poll loop with your network i/o because of this issue. This is why libuv falls back to using threads and synchronous i/o to provide "async" i/o.

MichaelMoser123 9y ago
you can do io_submit from one thread and then have some dedicated threads do io_getevent + event signalling (comes at a price though). Still all this event goodness is not good enough for things like infiniband and the brave world of zero copy networking.
- exDM69 9y ago
  Yeah, sure that kinda works but it won't play nice with having an event loop on network events. A typical high performance server has multiple worker threads waiting on events on a select/epoll and if you add in another thread (or several) waiting on io_getevent you end up in a rather contrived solution. You could use a pipe or eventfd or some other fd-based mechanism to wake up a worker thread on a filesystem event, but then you get two context switches (one to wake up io_getevent thread and another to wake up a worker). This carries a lot of indirect costs from jumping back and forth to the kernel. I am unaware of a solution that would make filesystem i/o and network i/o play nice in one event loop without adding some overhead.
derefr 9y ago
> Currently it's impossible (?) to have filesystem i/o in the same select/poll loop with your network i/o because of this issue. Hacky workaround: don't do filesystem i/o in the first place. Have the kernel serve an iSCSI target, then consume it over the loopback interface entirely in userspace. Your disk now has network semantics! (But you have to bring your own filesystem driver.) If you don't mind some extra abstraction overhead, you can skip past iSCSI and instead build a userspace NFS client lib into your app, consuming a loopback NFS share. s/NFS/9P/ if your disk server has in-kernel 9P. (Is there any use to these approaches? Yes: they're actually nearly-optimal ways to persist state from a unikernel running on a hypervisor, because all the disks are remote one way or another anyway. Erlang on Xen uses the 9P approach, for example.)

joosters 9y ago

AFAIK, there's no reason why an OS can't make filesystem I/O non-blocking with the standard read/write/select (or poll or whatever) system calls. The only tricky part (API-side) is that write() is defined to confirm that some data got written, or not. If you let write() become non-blocking, and deferred the I/O, perhaps by returning immediately with EAGAIN or some such response, the write is is an unknown state to the rest of the application. What data can then be expected from other read() calls on that file from the same process? The old data? The new? A mix? You'd have to drop the POSIX consistency guarantee. For most programs, this is never likely to be an issue, as they don't tend to be re-reading bits of a file that they are also writing to, so it's a shame there's no option to let the write() be non-blocking if requested. Issue 2 is failures: If the write, when it eventually happens, fails, you've got to have some way of passing that information back to the process. The solution for this seems easy enough; just let the FD become writeable again and the program will probably re-attempt the same write, at which time you can report the previous error. Finally, making open() calls non-blocking would be the real API changer. If you defer all of the file opening work and give the application a file descriptor before doing any I/O, then you can hit all kinds of errors (permissions problems, file not found, and so on). How do you pass them back to the program? You can make any future read() or write() calls fail but these syscalls would now have to return error codes that they aren't specified as actually being able to return (read can't return things like EEXIST, EACCESS and so on). So the API is inadequate for these situations. You can see how they could be extended to cover these cases though.

jzwinck 9y ago
File write could work the same as socket send. If the data cannot definitely be written now, write nothing and allow the application to try again, probably after waiting for readiness via poll or select.
hoytech 9y ago
See gpderetta's comment elsewhere in this thread: the page-cache is different than socket mbufs in some pretty significant ways.