5 ms·

The article explains how to use epoll() correctly to solve all the problems he raises. That's not 'fundamentally broken,' because it works. The worst you can sa

The article explains how to use epoll() correctly to solve all the problems he raises. That's not 'fundamentally broken,' because it works. The worst you can say is confusing and painful. Maybe that doesn't make good headlines, though. When you're dealing with shared resources (like a listening socket), and you are using threads, and you are trying to maximize performance with networking, confusing and painful is kind of the nature of the problem.

zzzcpan 10y ago
> The worst you can say is confusing and painful Everyone's approach towards these mechanisms is broken. Just don't treat it as a reliable notification mechanism, do your own scheduling using information from epoll/kqueue only as hints and everything will be fine.
- ktRolster 10y ago
  I admit, sometimes when I read these articles, I have trouble understanding why these people are having so much difficulty getting it to work. It's also hard for me to imagine a scenario where accept() takes longer than servicing a request and becomes a bottleneck. That is, why would you need multiple threads accepting on the same socket?
  - majke 10y ago
    Author here. I showed an example - short lived http 1.0 connections. I haven't done benchmarks but my hunch is that you can run accept at in low tens of thousands times per second from one CPU. If you have more than, say, 10k qps the accept() might well be the bottleneck. You can imagine having a box with 64 CPU's, doing short-lived connections, and being limited by accept() done on only one CPU since it doesn't scale. Second issue is cache locality. If you do accept() in one thread only, then you will need to move the new accept-ed client socket to another worker thread. Depending on details this might not be efficient - aRFS comes into mind. (but frankly, epoll alone won't help here, you need SO_REUSEPORT with SO_INCOMING_CPU). Even if you don't agree that scaling out accept is a real concern - that's missing the point. The point is: the epoll() model should take this into account and at least support this problem. Or loudly say that scaling out accept() with epoll is not possible. But neither things happened. Up till kernel 4.5 it was impossible to do it correctly, but undocumented, from 4.5 you can use the EPOLLEXCLUSIVE flag, which I feel is a hack.
    - ktRolster 10y ago
      I haven't done benchmarks but my hunch Your writing will be more convincing if you profile.
gizmo 10y ago
The linked interview with Bryan Cantrill gives a good explanation of why it's broken: a straightforward implementation behaves in a way you never want, with bizarre behavior where the kernel can wake up a thread with an accept() on a fd that has already been closed. linux should just have adopted kqueue or an IOCP model instead. https://youtu.be/l6XQUciI-Sc?t=3643 https://youtu.be/l6XQUciI-Sc?t=3643
- ktRolster 10y ago
  >accept() on a fd that has already been closed. Why are people doing that?
  - astrodust 10y ago
    Threads, like drugs, make you do strange and unpredictable things.
- dboreham 10y ago
  I believe at the time there was concern that Microsoft would sue for patent infringement relating to IOCP. Source: at the time I worked on high-performance server products and had discussions with various people active on the kernel side of the problem and encountered much head-shaking and muttering about patents, when I mentioned implementing IOCP.
  - aijeiy9X 10y ago
    Didn't Linus said that all he cared about is getting code back, with no concern over patent grants such as the one in gplv3? Head-shaking and muttering is all good but the kernel developers has had a long time to take an active stance against software patents and has consistently chosen not to do so. As a project they have chosen not to get political about patents, so this is the reality they have embraced.
    - dboreham 10y ago
      I think we're saying the same thing. fwiw there was an IOCP-like capability added to AIX, again supposedly because IBM did not have fears about patent infringement (likely due to cross-licensing arrangements).
seabee 10y ago
The most generous interpretation is that it can work - if you're careful, and if you're using a kernel from 2016. While I trust the author to do this (thankfully, as he's my coworker) there is a lot of Linux software that doesn't, even assuming it was updated in the last year and you're running something vaguely bleeding edge (not Debian).
- temac 10y ago
  A new Debian is about to be released, featuring Linux 4.9. Some people using a completely different kind of OS insist in continuing to use a version from 2009, instead of its more recent version. I guarantee you that this version from 2009 also has its share of dark APIs, some of which have been fixed in the modern version... Yet I've yet to find articles titled like if all of them are equally broken, while the content would precisely describe the caveats to do not-broken things on the capable recent releases.
wyldfire 10y ago
Don't we usually hide warts like this behind libuv or some other popular library? I wonder if we could put a sane fix into libc to fix these problems.
- orcy 10y ago
  As far as I know in libuv it is not possible to share socket in different event loops. But the point is while it might be possible to build an abstraction to fix the broken API, is not it would be better to have fixed API instead
baotiao 10y ago
Yes, exactly. when you deal with multi threads, not only epoll will cause some problems, but also global variable, memory, etc. global variable solved by mutex, but epoll solved by avoid using it to epoll_wait fds in multi threads.

zzzcpan 10y ago

> The worst you can say is confusing and painful Everyone's approach towards these mechanisms is broken. Just don't treat it as a reliable notification mechanism, do your own scheduling using information from epoll/kqueue only as hints and everything will be fine.

ktRolster 10y ago
I admit, sometimes when I read these articles, I have trouble understanding why these people are having so much difficulty getting it to work. It's also hard for me to imagine a scenario where accept() takes longer than servicing a request and becomes a bottleneck. That is, why would you need multiple threads accepting on the same socket?
- majke 10y ago
  Author here. I showed an example - short lived http 1.0 connections. I haven't done benchmarks but my hunch is that you can run accept at in low tens of thousands times per second from one CPU. If you have more than, say, 10k qps the accept() might well be the bottleneck. You can imagine having a box with 64 CPU's, doing short-lived connections, and being limited by accept() done on only one CPU since it doesn't scale. Second issue is cache locality. If you do accept() in one thread only, then you will need to move the new accept-ed client socket to another worker thread. Depending on details this might not be efficient - aRFS comes into mind. (but frankly, epoll alone won't help here, you need SO_REUSEPORT with SO_INCOMING_CPU). Even if you don't agree that scaling out accept is a real concern - that's missing the point. The point is: the epoll() model should take this into account and at least support this problem. Or loudly say that scaling out accept() with epoll is not possible. But neither things happened. Up till kernel 4.5 it was impossible to do it correctly, but undocumented, from 4.5 you can use the EPOLLEXCLUSIVE flag, which I feel is a hack.
  - ktRolster 10y ago
    I haven't done benchmarks but my hunch Your writing will be more convincing if you profile.

gizmo 10y ago

The linked interview with Bryan Cantrill gives a good explanation of why it's broken: a straightforward implementation behaves in a way you never want, with bizarre behavior where the kernel can wake up a thread with an accept() on a fd that has already been closed. linux should just have adopted kqueue or an IOCP model instead. https://youtu.be/l6XQUciI-Sc?t=3643 https://youtu.be/l6XQUciI-Sc?t=3643

ktRolster 10y ago
>accept() on a fd that has already been closed. Why are people doing that?
- astrodust 10y ago
  Threads, like drugs, make you do strange and unpredictable things.
dboreham 10y ago
I believe at the time there was concern that Microsoft would sue for patent infringement relating to IOCP. Source: at the time I worked on high-performance server products and had discussions with various people active on the kernel side of the problem and encountered much head-shaking and muttering about patents, when I mentioned implementing IOCP.
- aijeiy9X 10y ago
  Didn't Linus said that all he cared about is getting code back, with no concern over patent grants such as the one in gplv3? Head-shaking and muttering is all good but the kernel developers has had a long time to take an active stance against software patents and has consistently chosen not to do so. As a project they have chosen not to get political about patents, so this is the reality they have embraced.
  - dboreham 10y ago
    I think we're saying the same thing. fwiw there was an IOCP-like capability added to AIX, again supposedly because IBM did not have fears about patent infringement (likely due to cross-licensing arrangements).

seabee 10y ago

The most generous interpretation is that it can work - if you're careful, and if you're using a kernel from 2016. While I trust the author to do this (thankfully, as he's my coworker) there is a lot of Linux software that doesn't, even assuming it was updated in the last year and you're running something vaguely bleeding edge (not Debian).

temac 10y ago
A new Debian is about to be released, featuring Linux 4.9. Some people using a completely different kind of OS insist in continuing to use a version from 2009, instead of its more recent version. I guarantee you that this version from 2009 also has its share of dark APIs, some of which have been fixed in the modern version... Yet I've yet to find articles titled like if all of them are equally broken, while the content would precisely describe the caveats to do not-broken things on the capable recent releases.

wyldfire 10y ago

Don't we usually hide warts like this behind libuv or some other popular library? I wonder if we could put a sane fix into libc to fix these problems.

orcy 10y ago
As far as I know in libuv it is not possible to share socket in different event loops. But the point is while it might be possible to build an abstraction to fix the broken API, is not it would be better to have fixed API instead

baotiao 10y ago

Yes, exactly. when you deal with multi threads, not only epoll will cause some problems, but also global variable, memory, etc. global variable solved by mutex, but epoll solved by avoid using it to epoll_wait fds in multi threads.