4 ms·

Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() so this doesn't actually help read from disks (even when they're

by geocar 9y ago

Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() so this doesn't actually help read from disks (even when they're backed by network/shared storage). That's why you often see this kind of seeming "nonsense" in HPC startup scripts: for x in *; do cat "$x" & done | pipeline ... Furthermore, in a networked-environment, select()/poll()/epoll() actually decrease concurrency compared to threads because now you need another system call to go fetch the data from the kernel. If I've got the CPUs I'd rather use threads. In the Windows model (in contrast), the programmer says "when this data is available (from this socket, file, etc), put it here (into this memory buffer)" and then waits for the kernel to do something. As a result, you skip that extra syscall -- but you also skip any other read you might have needed to do for data that's arrived at (roughly) the same time. AIO[1] is potentially similar, but it never seems to benchmark well in my experience. [1]: http://man7.org/linux/man-pages/man7/aio.7.html http://man7.org/linux/man-pages/man7/aio.7.html

FooBarWidget 9y ago
What is surprising to me is that it is 2017 and nobody has attempted to solve these problems. Maybe the current mechanisms, with all their faults, are considered good enough?
- AstralStorm 9y ago
  Generally epoll and event io are good enough for millions of simultaneous files or sockets. Nobody tells you to use direct read or write instead of mmap either.
  - cturner 9y ago
    Mmap - I can't think of a scenario where it improves over read or write in the context discussed here: async access. Mmap is synchronous, and you can't do select/poll on it.
    - geocar 9y ago
      https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt
      - cturner 9y ago
        Not sure why you were downvoted. That's some seriously hairy stuff there. I reckon I'd move to building my app as a kernel module more readily than trying to make a robust async arrangement using that mechanism :) Call me demanding, but I think we should can reasonably expect easier access to async from our OS than this mechanism offers. On that topic of which.. how's the future of operating systems coming along?
- dom0 9y ago
  Other systems (Windows and I think Solaris, but some more obscure systems likely do this as well) can do asynchronous disk IO, but it's not perfect, either. I think the main reason Linux doesn't really do it is that asynchronous IO is an intrusive change and there would be just too much work to implement it in all the file systems etc.; i.e. "not worth it". Many applications which are not OK with synchronous disk I/O seem to find thread pools good enough: reasonably easy to implement, reasonably portable, usually performs ok.
  - adrianratnapala 9y ago
    I only half buy the "not worth it". Traditional unix, including Linux pretends to support poll() etc for file I/O (i.e. it supports the interfaces, but the reality is synchronous). The reason for this is that an application that expects async I/O can kinda-sorta-work with real files -- and "kinda sorta" will not be too bad because local disks are fast enough to paper over any problem. But then if the disks aren't local, why not actually make non-blocking I/O work as advertised? Actually I can guess the answere: there are 101 corner cases that mean I can't neatly separate out the apps design for noblocking I/O. And that's why I half buy the argument. But only half.
- mfukar 9y ago
  Those problems are already solved by async I/O in platforms other than Linux.
- geocar 9y ago
  I think that's unfair: A lot of people have attempted to solve these problems(), and the Windows Way™ has it's own problems. The real answer is probably between elastic infrastructure and the fact that computers got faster and cheaper, and so -- at least for a while the number of people needing solutions here went down. https://lwn.net/Articles/219954/ https://lwn.net/Articles/219954/ * https://lwn.net/Articles/223899/ https://lwn.net/Articles/223899/ * https://lwn.net/Articles/221913/ https://lwn.net/Articles/221913/ * https://lwn.net/Articles/316806/ https://lwn.net/Articles/316806/ * https://lwn.net/Articles/612038/ https://lwn.net/Articles/612038/ * https://lwn.net/Articles/671658/ https://lwn.net/Articles/671658/ * http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt * http://ogris.de/howtos/splice.html http://ogris.de/howtos/splice.html And so on. However a few companies have seen real commercial benefit from actually analysing their data, so we're seeing a lot of "big data" interest. Some of these companies have noticed that a bunch of hadoop/mongo/whatever boxes can't even approach the performance of one "big" box tuned right, so we're seeing somewhat of a resurgence in interest here.
  - int_19h 9y ago
    > and the Windows Way™ has it's own problems Could you expand on that? (I'm assuming that by the "Windows Way", you mean I/O completion ports.)
gpderetta 9y ago
> Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() There is no other choice really. Let's say you want to read a specific page from a file. No page is currently in the kernel buffers. If poll where to return non-ready (and remember that poll has no idea which pages you are interested in), what action, within posix, would you take to change it? poll and friends just do not make sense for plain file descriptors. In linux, splice add an interesting twist, but it is still awkward to use.
- geocar 9y ago
  > Let's say you want to read a specific page from a file. pread() is an unusual case. I'd be happy if poll() simply indicated read() would succeed, and I'm happy doing an lseek() before every read(). However a pread() with O_NONBLOCK could return EAGAIN and poll() would then know what pages I'm interested in. > what action, within posix, would you take to change it? The general (POSIX-compatible) strategy (as I indicated) has been to fork a thread off to handle reading so that my application is notified when the data is in the fifo. This works for me when I'm reading (say) five files from a slow (or remote) storage, but it's not ideal since it causes a copy.
  - gpderetta 9y ago
    poll can't guarantee that the next read will succeed because first of all doesn't know how many pages you are interested in reading and any relevant page in the page cache might be evicted between the poll and the read. Also having pread, which is designed for concurrency and scalability, carry state around for poll (an unbounded amount of state btw: you can have multiple threads issuing concurrent preads on the same fd) makes it completely pointless. There are linux patches floating to add flags to preadv2 so that on a page not present it returns EAGAIN, so that you can optimistically do a sync read an fall back to a thread pool in the slow case. It still doesn't involve poll and friend.
    - geocar 9y ago
      poll() can't guarantee that the next read will succeed anyway, because something else might drain the socket or fifo (e.g. another thread). Programmers simply use O_NONBLOCK anyway and use poll() is wait until the process should try again.
crest 9y ago
In theory FreeBSD supports a sane aio_read/write/fsync() implementation by extending sigevent to include the generation of kqueue events, but there are no aio_open/close/fstat/rename() syscalls and if you need a thread pool to deal with those remaining blocking file system calls there are few advantages to using the in-kernel thread pool implementing the aio_*() syscalls. An other problem is that filesystem aio calls are limited to local file systems, because accesses to network file systems could block indefinitely starving local file system accesses by blocking all threads in the pool.

by geocar 9y ago

FooBarWidget 9y ago

What is surprising to me is that it is 2017 and nobody has attempted to solve these problems. Maybe the current mechanisms, with all their faults, are considered good enough?

AstralStorm 9y ago
Generally epoll and event io are good enough for millions of simultaneous files or sockets. Nobody tells you to use direct read or write instead of mmap either.
- cturner 9y ago
  Mmap - I can't think of a scenario where it improves over read or write in the context discussed here: async access. Mmap is synchronous, and you can't do select/poll on it.
  - geocar 9y ago
    https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt
    - cturner 9y ago
      Not sure why you were downvoted. That's some seriously hairy stuff there. I reckon I'd move to building my app as a kernel module more readily than trying to make a robust async arrangement using that mechanism :) Call me demanding, but I think we should can reasonably expect easier access to async from our OS than this mechanism offers. On that topic of which.. how's the future of operating systems coming along?
dom0 9y ago
Other systems (Windows and I think Solaris, but some more obscure systems likely do this as well) can do asynchronous disk IO, but it's not perfect, either. I think the main reason Linux doesn't really do it is that asynchronous IO is an intrusive change and there would be just too much work to implement it in all the file systems etc.; i.e. "not worth it". Many applications which are not OK with synchronous disk I/O seem to find thread pools good enough: reasonably easy to implement, reasonably portable, usually performs ok.
- adrianratnapala 9y ago
  I only half buy the "not worth it". Traditional unix, including Linux pretends to support poll() etc for file I/O (i.e. it supports the interfaces, but the reality is synchronous). The reason for this is that an application that expects async I/O can kinda-sorta-work with real files -- and "kinda sorta" will not be too bad because local disks are fast enough to paper over any problem. But then if the disks aren't local, why not actually make non-blocking I/O work as advertised? Actually I can guess the answere: there are 101 corner cases that mean I can't neatly separate out the apps design for noblocking I/O. And that's why I half buy the argument. But only half.
mfukar 9y ago
Those problems are already solved by async I/O in platforms other than Linux.
geocar 9y ago
I think that's unfair: A lot of people have attempted to solve these problems(), and the Windows Way™ has it's own problems. The real answer is probably between elastic infrastructure and the fact that computers got faster and cheaper, and so -- at least for a while the number of people needing solutions here went down. https://lwn.net/Articles/219954/ https://lwn.net/Articles/219954/ * https://lwn.net/Articles/223899/ https://lwn.net/Articles/223899/ * https://lwn.net/Articles/221913/ https://lwn.net/Articles/221913/ * https://lwn.net/Articles/316806/ https://lwn.net/Articles/316806/ * https://lwn.net/Articles/612038/ https://lwn.net/Articles/612038/ * https://lwn.net/Articles/671658/ https://lwn.net/Articles/671658/ * http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt * http://ogris.de/howtos/splice.html http://ogris.de/howtos/splice.html And so on. However a few companies have seen real commercial benefit from actually analysing their data, so we're seeing a lot of "big data" interest. Some of these companies have noticed that a bunch of hadoop/mongo/whatever boxes can't even approach the performance of one "big" box tuned right, so we're seeing somewhat of a resurgence in interest here.
- int_19h 9y ago
  > and the Windows Way™ has it's own problems Could you expand on that? (I'm assuming that by the "Windows Way", you mean I/O completion ports.)

gpderetta 9y ago

> Most unixes (including Linux) always report files to be readable under select()/poll()/epoll() There is no other choice really. Let's say you want to read a specific page from a file. No page is currently in the kernel buffers. If poll where to return non-ready (and remember that poll has no idea which pages you are interested in), what action, within posix, would you take to change it? poll and friends just do not make sense for plain file descriptors. In linux, splice add an interesting twist, but it is still awkward to use.

geocar 9y ago
> Let's say you want to read a specific page from a file. pread() is an unusual case. I'd be happy if poll() simply indicated read() would succeed, and I'm happy doing an lseek() before every read(). However a pread() with O_NONBLOCK could return EAGAIN and poll() would then know what pages I'm interested in. > what action, within posix, would you take to change it? The general (POSIX-compatible) strategy (as I indicated) has been to fork a thread off to handle reading so that my application is notified when the data is in the fifo. This works for me when I'm reading (say) five files from a slow (or remote) storage, but it's not ideal since it causes a copy.
- gpderetta 9y ago
  poll can't guarantee that the next read will succeed because first of all doesn't know how many pages you are interested in reading and any relevant page in the page cache might be evicted between the poll and the read. Also having pread, which is designed for concurrency and scalability, carry state around for poll (an unbounded amount of state btw: you can have multiple threads issuing concurrent preads on the same fd) makes it completely pointless. There are linux patches floating to add flags to preadv2 so that on a page not present it returns EAGAIN, so that you can optimistically do a sync read an fall back to a thread pool in the slow case. It still doesn't involve poll and friend.
  - geocar 9y ago
    poll() can't guarantee that the next read will succeed anyway, because something else might drain the socket or fifo (e.g. another thread). Programmers simply use O_NONBLOCK anyway and use poll() is wait until the process should try again.

crest 9y ago

In theory FreeBSD supports a sane aio_read/write/fsync() implementation by extending sigevent to include the generation of kqueue events, but there are no aio_open/close/fstat/rename() syscalls and if you need a thread pool to deal with those remaining blocking file system calls there are few advantages to using the in-kernel thread pool implementing the aio_*() syscalls. An other problem is that filesystem aio calls are limited to local file systems, because accesses to network file systems could block indefinitely starving local file system accesses by blocking all threads in the pool.