4 ms·

It would be interesting to compare this with coroutines/fibers, and it would also be interesting to know how latency is affected. In my experience, thread pools

It would be interesting to compare this with coroutines/fibers, and it would also be interesting to know how latency is affected. In my experience, thread pools introduce latency.

josteink 9y ago
> In my experience, thread pools introduce latency. The headline uses a the word "performance" which is rather ambiguous. Performance has many aspects: Latency, through-put, resource-usage (doing more with less), etc etc. A more specific headline wouldn't hurt.
- ogrisel 9y ago
  In the benchmarks reported in the article, both through-put and latency were significantly improved by the new thread pool system.
- Cursuviam 9y ago
  Being specific about the area being improved (blocking operations with the example of file reads) could be nice.
nemanjaboric 9y ago
Trick is that nginx uses thread pools for non-blocking reading on the "fast" devices - you can't use coroutines/fibers, because on Linux is not reliably possible to read the regular file in a non-blocking way, as far as I'm aware, so that means that `read` in a single fiber would block entire thread until the data is fetched from the disk. In the storage engine I'm developing/maintaining, I've been using fibers, for both network and disk IO, until people started noticing that, if number of the client grows, and if they ask a lot of data (generating a lot of disk traffic), the performance drops severely. I've moved to the thread pools, dispatching my disk IO operations there (although it's much problematic for the reads, as writes would be performed by kernel anyway, `fsync` and `close` would be operations that you still want to do in the separate thread): https://github.com/sociomantic-tsunami/dlsnode/blob/master/src/dlsnode/util/aio/AsyncIO.d https://github.com/sociomantic-tsunami/dlsnode/blob/master/s... edit: used right link for the freshly opensourced repo.
- vbezhenar 9y ago
  Why files can't be read without blocking with Linux? There's certainly an API for this.
  - signa11 9y ago
    > Why files can't be read without blocking with Linux? There's certainly an API for this. i guess you are referring to aio(7) ? posix aio on linux is implemented in glibc, and doesn't scale in presence of multiple threads. which is probably what is alluded to...
  - nemanjaboric 9y ago
    Which API are you referring to? If you're referring to `aio_` that's just glibc's user-space thread-pool emulation of the POSIX's asynchronous IO + many operations needed for the truly async IO are not supported (such as `stat` and `open`). There's `io_` syscalls family, but that's not in the ready-to-use state yet, judging by the manpages and various patches being submitted every month.
    - vbezhenar 9y ago
      I'm referring to `fcntl(fd, F_SETFL, flags | O_NONBLOCK)`. `stat`, `open` are blocking, yes, didn't think about that. Also according to [1] file operations will always block, even in non-blocking mode, so I was wrong, sorry. 1: https://www.remlab.net/op/nonblock.shtml https://www.remlab.net/op/nonblock.shtml
      - signa11 9y ago
        > ... Also according to [1] file operations will always block ... not quite :) look at aio(7)
  - biokoda 9y ago
    io_ syscalls only really work on XFS and require O_DIRECT which means kernel is not caching anything. Unsuitable for web servers.
dunkelheit 9y ago
Fibers and thread pools are not exclusive - you can schedule fibers onto threads. Nginx already uses event-based state-machine approach so context switch is even cheaper than with fibers (at the price of ugly code). What they talk about in the article is that they offload potentially blocking operations to thread pools. That's not about networking, which is already done asynchronously, but about e.g. reading from a file not present in the page cache. Note that detecting that the needed bytes are already in the page cache and thus the read wouldn't block and should not be offloaded is tricky. As far as I can see they haven't done that yet so this new feature (edit: not so new, the post is from 2015) is not without downsides.
- continuations 9y ago
  Why can Linux do async network IO but not async file IO? Seems like they would be similar. What's so special about file IO?
  - crest 9y ago
    When non-blocking socket I/O was implemented the disk was considered to be too fast to warrant the same kind of treatment. At the API level the different is that sockets, pipes and ttys are have an additional "ready for use" state while files are always ready, because the kernel can do something to access them even if it means blocking the thread while the kernel waits for the disks to respond.
    - nemanjaboric 9y ago
      Yes, the trick is that disk device is considered "fast", so it's not selectable - it's always ready to read/write, which is a lie. I _feel_ (and think, I don't have any data to prove this) the main problem here is the file system layer, which may or may not need to block _after_ the operation is performed, and this complexity doesn't occur with sockets/pipes. Having a always-ready state on regular files is a problem since Linux's non-blocking is going around readiness. On Windows (and I believe on Solaris/FreeBSD), completeness model is in place, so you schedule an operation, kernel does _everything_ and then you're resumed upon completion of IO.

It would be interesting to compare this with coroutines/fibers, and it would also be interesting to know how latency is affected. In my experience, thread pools introduce latency.

josteink 9y ago

> In my experience, thread pools introduce latency. The headline uses a the word "performance" which is rather ambiguous. Performance has many aspects: Latency, through-put, resource-usage (doing more with less), etc etc. A more specific headline wouldn't hurt.

ogrisel 9y ago
In the benchmarks reported in the article, both through-put and latency were significantly improved by the new thread pool system.
Cursuviam 9y ago
Being specific about the area being improved (blocking operations with the example of file reads) could be nice.

nemanjaboric 9y ago

Trick is that nginx uses thread pools for non-blocking reading on the "fast" devices - you can't use coroutines/fibers, because on Linux is not reliably possible to read the regular file in a non-blocking way, as far as I'm aware, so that means that `read` in a single fiber would block entire thread until the data is fetched from the disk. In the storage engine I'm developing/maintaining, I've been using fibers, for both network and disk IO, until people started noticing that, if number of the client grows, and if they ask a lot of data (generating a lot of disk traffic), the performance drops severely. I've moved to the thread pools, dispatching my disk IO operations there (although it's much problematic for the reads, as writes would be performed by kernel anyway, `fsync` and `close` would be operations that you still want to do in the separate thread): https://github.com/sociomantic-tsunami/dlsnode/blob/master/src/dlsnode/util/aio/AsyncIO.d https://github.com/sociomantic-tsunami/dlsnode/blob/master/s... edit: used right link for the freshly opensourced repo.

vbezhenar 9y ago
Why files can't be read without blocking with Linux? There's certainly an API for this.
- signa11 9y ago
  > Why files can't be read without blocking with Linux? There's certainly an API for this. i guess you are referring to aio(7) ? posix aio on linux is implemented in glibc, and doesn't scale in presence of multiple threads. which is probably what is alluded to...
- nemanjaboric 9y ago
  Which API are you referring to? If you're referring to `aio_` that's just glibc's user-space thread-pool emulation of the POSIX's asynchronous IO + many operations needed for the truly async IO are not supported (such as `stat` and `open`). There's `io_` syscalls family, but that's not in the ready-to-use state yet, judging by the manpages and various patches being submitted every month.
  - vbezhenar 9y ago
    I'm referring to `fcntl(fd, F_SETFL, flags | O_NONBLOCK)`. `stat`, `open` are blocking, yes, didn't think about that. Also according to [1] file operations will always block, even in non-blocking mode, so I was wrong, sorry. 1: https://www.remlab.net/op/nonblock.shtml https://www.remlab.net/op/nonblock.shtml
    - signa11 9y ago
      > ... Also according to [1] file operations will always block ... not quite :) look at aio(7)
- biokoda 9y ago
  io_ syscalls only really work on XFS and require O_DIRECT which means kernel is not caching anything. Unsuitable for web servers.

dunkelheit 9y ago

Fibers and thread pools are not exclusive - you can schedule fibers onto threads. Nginx already uses event-based state-machine approach so context switch is even cheaper than with fibers (at the price of ugly code). What they talk about in the article is that they offload potentially blocking operations to thread pools. That's not about networking, which is already done asynchronously, but about e.g. reading from a file not present in the page cache. Note that detecting that the needed bytes are already in the page cache and thus the read wouldn't block and should not be offloaded is tricky. As far as I can see they haven't done that yet so this new feature (edit: not so new, the post is from 2015) is not without downsides.

continuations 9y ago
Why can Linux do async network IO but not async file IO? Seems like they would be similar. What's so special about file IO?
- crest 9y ago
  When non-blocking socket I/O was implemented the disk was considered to be too fast to warrant the same kind of treatment. At the API level the different is that sockets, pipes and ttys are have an additional "ready for use" state while files are always ready, because the kernel can do something to access them even if it means blocking the thread while the kernel waits for the disks to respond.
  - nemanjaboric 9y ago
    Yes, the trick is that disk device is considered "fast", so it's not selectable - it's always ready to read/write, which is a lie. I _feel_ (and think, I don't have any data to prove this) the main problem here is the file system layer, which may or may not need to block _after_ the operation is performed, and this complexity doesn't occur with sockets/pipes. Having a always-ready state on regular files is a problem since Linux's non-blocking is going around readiness. On Windows (and I believe on Solaris/FreeBSD), completeness model is in place, so you schedule an operation, kernel does _everything_ and then you're resumed upon completion of IO.