7 ms·

Epoll is fundamentally broken (2017)

81 points by justin_ 4y ago

dxuh 4y ago
I have used select and poll before and sort of skipped to io_uring for a recent project, which also doesn't do too well with multiple threads (you have to use multiple rings and do everything else yourself). It's a shame that there is no obvious, relatively easy to use async IO mechanism on Linux that you can use from multiple threads without getting yourself in trouble. Reading more about kqueue it looks a lot like something that would solve it. Why was it not ported to Linux? Would it have been too hard to integrate? The linked article is great btw. Very informative and concise!
- kasabali 4y ago
  > Why was it not ported to Linux? NIH
  - thatcherc 4y ago
    National Institutes of Health?
    - tyingq 4y ago
      not invented here
    - cassepipe 4y ago
      Not Invented Here, old chap
- ilyt 4y ago
  > I have used select and poll before and sort of skipped to io_uring for a recent project, which also doesn't do too well with multiple threads (you have to use multiple rings and do everything else yourself Wouldn't that be on purpose ? Coordination requires more cpu cycles and so cuts on max performance > Why was it not ported to Linux? We've been asking since epoll got introduced. But here is some context: https://lwn.net/Articles/431297/ https://lwn.net/Articles/431297/
- LaLaLand122 4y ago
  > you have to use multiple rings and do everything else yourself Since you are supposed to use liburing, not the kernel interface directly, I guess somebody could add multithreading "support" to it. Or at least add documentation/examples of the most common/performant options: https://github.com/shuveb/loti/issues/4 https://github.com/shuveb/loti/issues/4 AFAIR Windows IOCP handles multithreading by: - Handling locking at kernel level, the syscall is thread safe - Making it LIFO, to keep things in the same threads, to have a decent cache behaviour. It's as simple as it gets.
rwaksmunski 4y ago
Linux was the last to implement "events", they had IO completion ports and kqueue to learn from. They decided to implement something broken and document it as such, it was right in the man page. I find that astonishing. That reminds me, dnotify and inotify are probably still there. I'm bitter because I've had to port a clean kqueue event loop to Linux as a younger programmer and it was a dreadful experience.
- nextaccountic 4y ago
  Is inotify broken? What should be used instead of it to watch for filesystem events?
  - rwaksmunski 4y ago
    I don't know man, I've given up and handed it over to someone more senior. Their port was half the performance and the project was killed. We kept on using FreeBSD for the product.
    - nextaccountic 4y ago
      Is it broken regarding to correctness or just performance? I mean, does inotify have some obscure race condition or drop events or duplicate events or something
      - rwaksmunski 4y ago
        > Is it broken regarding to correctness or just performance? Yes. This blog entry explains it much better then I could: https://wingolog.org/archives/2018/05/21/correct-or-inotify-pick-one https://wingolog.org/archives/2018/05/21/correct-or-inotify-...
    - Beltalowda 4y ago
      For watching filesystem events FreeBSD's kqueue is much more difficult to use correctly though, and in my experience it certainly doesn't outperform inotify. inotify (and now, fanotify) are specifically designed to watch for filesystem events, whereas kqueue is a generic event system used for many things. In many ways that's better, but it this lack of specialisation also comes with some drawbacks.
adontz 4y ago
My experience of epoll is that it's for single process, single thread network I/O mostly. If you want multithreaded network server use SO_REUSEPORT, and one epoll per thread. Everything else is either broken, overcomplicated or slow.
- wbl 4y ago
  Or use IOCP on Windows or kqueue on FreeBSD. IOCP is particularly interesting as it will interact with the scheduler and release more threads if ones get blocked, trying to right-size CPU usage.
- mritun 4y ago
  This is the correct answer (on Linux) and the article dismisses it after lightly touching on it’s correctness. In a multi-threaded environment someone needs to pay the cost of synchronization if the entire event-queue is loadbalanced. If you don’t want the events to be trivially load-balanced and want the events from one fd to be delivered to a single worker, then it’s way better to use SO_REUSEPORT and get it right from the get go. Expecting kernel to solve a problem of user-space’s making is asking for trouble - can be done, but the edge cases will sink your project!
  - naasking 4y ago
    > Expecting kernel to solve a problem of user-space’s making is asking for trouble These are problems the kernel has introduced. I'm not sure you've read article carefully enough.
- jbluepolarbear 4y ago
  This is what I was thinking reading this. Why are they reading the the same buffer in 2 threads. This problem will always happen if reading the same buffer in 2 threads.
  - naasking 4y ago
    They are not. This happens for any fd, like a socket waiting for a connection.
samsquire 4y ago
I wrote an epoll echo server that multiplexes multiple network connections over threads (multiple users per thread) https://github.com/samsquire/epoll-server https://github.com/samsquire/epoll-server I also have a 1:M:N (1 scheduler thread, M kernel threads and N lightweight green threads) multithreaded userspace scheduler which multiplexes lightweight threads onto kernel threads and can preempt hot loops with minimal overhead. I rely on the fact that you can change the looping variable from another thread if you use a structure. Preemptive interruption is very useful for the illusion of multitasking. That's why I call it a userspace scheduler. https://GitHub.com/samsquire/preemptible-thread https://GitHub.com/samsquire/preemptible-thread I think the epoll-server which is kind of similar to what libuv does and the userspace scheduler could be combined into an application server. I also wrote a multithreaded actor implementation in Java. Threads can communicate with each other between 60 million - 100 million messages a second. The epoll-server uses a multiconsumer multiproducer lockless RingBuffer. https://GitHub.com/samsquire/multicersion-concurrency-control https://GitHub.com/samsquire/multicersion-concurrency-contro... I think the core fundamentals of building a performant application server should be done once and reused for each application. I want to also split the threading used by recv and send of a socket so that we have a 1:RecvKernelThread SendKernelThread with 1 RK+SK assigned to Socket:N scheduling (1 scheduler thread, 1 assigned Recv thread, 1 assigned send thread per socket). So you can send while you receive and receive while you send. True multiplexing! We can decouple CPU and IO completely with threading.

81 points by justin_ 4y ago

dxuh 4y ago

I have used select and poll before and sort of skipped to io_uring for a recent project, which also doesn't do too well with multiple threads (you have to use multiple rings and do everything else yourself). It's a shame that there is no obvious, relatively easy to use async IO mechanism on Linux that you can use from multiple threads without getting yourself in trouble. Reading more about kqueue it looks a lot like something that would solve it. Why was it not ported to Linux? Would it have been too hard to integrate? The linked article is great btw. Very informative and concise!

kasabali 4y ago
> Why was it not ported to Linux? NIH
- thatcherc 4y ago
  National Institutes of Health?
  - tyingq 4y ago
    not invented here
  - cassepipe 4y ago
    Not Invented Here, old chap
ilyt 4y ago
> I have used select and poll before and sort of skipped to io_uring for a recent project, which also doesn't do too well with multiple threads (you have to use multiple rings and do everything else yourself Wouldn't that be on purpose ? Coordination requires more cpu cycles and so cuts on max performance > Why was it not ported to Linux? We've been asking since epoll got introduced. But here is some context: https://lwn.net/Articles/431297/ https://lwn.net/Articles/431297/
LaLaLand122 4y ago
> you have to use multiple rings and do everything else yourself Since you are supposed to use liburing, not the kernel interface directly, I guess somebody could add multithreading "support" to it. Or at least add documentation/examples of the most common/performant options: https://github.com/shuveb/loti/issues/4 https://github.com/shuveb/loti/issues/4 AFAIR Windows IOCP handles multithreading by: - Handling locking at kernel level, the syscall is thread safe - Making it LIFO, to keep things in the same threads, to have a decent cache behaviour. It's as simple as it gets.

rwaksmunski 4y ago

Linux was the last to implement "events", they had IO completion ports and kqueue to learn from. They decided to implement something broken and document it as such, it was right in the man page. I find that astonishing. That reminds me, dnotify and inotify are probably still there. I'm bitter because I've had to port a clean kqueue event loop to Linux as a younger programmer and it was a dreadful experience.

nextaccountic 4y ago
Is inotify broken? What should be used instead of it to watch for filesystem events?
- rwaksmunski 4y ago
  I don't know man, I've given up and handed it over to someone more senior. Their port was half the performance and the project was killed. We kept on using FreeBSD for the product.
  - nextaccountic 4y ago
    Is it broken regarding to correctness or just performance? I mean, does inotify have some obscure race condition or drop events or duplicate events or something
    - rwaksmunski 4y ago
      > Is it broken regarding to correctness or just performance? Yes. This blog entry explains it much better then I could: https://wingolog.org/archives/2018/05/21/correct-or-inotify-pick-one https://wingolog.org/archives/2018/05/21/correct-or-inotify-...
  - Beltalowda 4y ago
    For watching filesystem events FreeBSD's kqueue is much more difficult to use correctly though, and in my experience it certainly doesn't outperform inotify. inotify (and now, fanotify) are specifically designed to watch for filesystem events, whereas kqueue is a generic event system used for many things. In many ways that's better, but it this lack of specialisation also comes with some drawbacks.

adontz 4y ago

My experience of epoll is that it's for single process, single thread network I/O mostly. If you want multithreaded network server use SO_REUSEPORT, and one epoll per thread. Everything else is either broken, overcomplicated or slow.

wbl 4y ago
Or use IOCP on Windows or kqueue on FreeBSD. IOCP is particularly interesting as it will interact with the scheduler and release more threads if ones get blocked, trying to right-size CPU usage.
mritun 4y ago
This is the correct answer (on Linux) and the article dismisses it after lightly touching on it’s correctness. In a multi-threaded environment someone needs to pay the cost of synchronization if the entire event-queue is loadbalanced. If you don’t want the events to be trivially load-balanced and want the events from one fd to be delivered to a single worker, then it’s way better to use SO_REUSEPORT and get it right from the get go. Expecting kernel to solve a problem of user-space’s making is asking for trouble - can be done, but the edge cases will sink your project!
- naasking 4y ago
  > Expecting kernel to solve a problem of user-space’s making is asking for trouble These are problems the kernel has introduced. I'm not sure you've read article carefully enough.
jbluepolarbear 4y ago
This is what I was thinking reading this. Why are they reading the the same buffer in 2 threads. This problem will always happen if reading the same buffer in 2 threads.
- naasking 4y ago
  They are not. This happens for any fd, like a socket waiting for a connection.

samsquire 4y ago

I wrote an epoll echo server that multiplexes multiple network connections over threads (multiple users per thread) https://github.com/samsquire/epoll-server https://github.com/samsquire/epoll-server I also have a 1:M:N (1 scheduler thread, M kernel threads and N lightweight green threads) multithreaded userspace scheduler which multiplexes lightweight threads onto kernel threads and can preempt hot loops with minimal overhead. I rely on the fact that you can change the looping variable from another thread if you use a structure. Preemptive interruption is very useful for the illusion of multitasking. That's why I call it a userspace scheduler. https://GitHub.com/samsquire/preemptible-thread https://GitHub.com/samsquire/preemptible-thread I think the epoll-server which is kind of similar to what libuv does and the userspace scheduler could be combined into an application server. I also wrote a multithreaded actor implementation in Java. Threads can communicate with each other between 60 million - 100 million messages a second. The epoll-server uses a multiconsumer multiproducer lockless RingBuffer. https://GitHub.com/samsquire/multicersion-concurrency-control https://GitHub.com/samsquire/multicersion-concurrency-contro... I think the core fundamentals of building a performant application server should be done once and reused for each application. I want to also split the threading used by recv and send of a socket so that we have a 1:RecvKernelThread SendKernelThread with 1 RK+SK assigned to Socket:N scheduling (1 scheduler thread, 1 assigned Recv thread, 1 assigned send thread per socket). So you can send while you receive and receive while you send. True multiplexing! We can decouple CPU and IO completely with threading.