4 ms·

Interesting writeup, seems like this only applies to people writing epoll abstraction layers though. Everybody else is able to avoid this bug by deregistering t

by jgh 10y ago

Interesting writeup, seems like this only applies to people writing epoll abstraction layers though. Everybody else is able to avoid this bug by deregistering the fds before calling close. So I'm not sure calling it "fundamentally broken" is necessarily accurate.

bluejekyll 10y ago
Error prone? It definitely creates situations where bugs crop up. The example of close() is actually quite important, and something which is perhaps unexpected until you hit that bug and spend hours or days tracking it down. Definitely has happened to me.
joosters 10y ago
Yeah, I think 'fundamentally broken' is going into clickbait territory - these are API calls that are successfully used by a huge number of programs. epoll() is the most recent of these multiplexer syscalls and it's been around for a decade or so. The underlying message is that there are corner cases that exist and should be thought about, but that is true of most syscalls. The epoll() call itself is a strange API. As the series of articles says, it was introduced to solve the problem of the kernel having to register and unregister every listed FD on every invocation. However, as a trade-off, your program now needs to make a syscall each time it wants to add/remove an FD. So where you once had a single poll() call per event loop, you now have to fire off extra syscalls for each single FD on each new connection. Compare this to the API of FreeBSD's kqueue - it is another solution to avoid the linear costs of select()/poll(), but instead of making a syscall for registering/unregistering each new FD, kevent() is called with a list of FDs that you want to change. So an event loop using kqueue consists of just one syscall per loop.
- zrm 10y ago
  > Compare this to the API of FreeBSD's kqueue - it is another solution to avoid the linear costs of select()/poll(), but instead of making a syscall for registering/unregistering each new FD, kevent() is called with a list of FDs that you want to change. So an event loop using kqueue consists of just one syscall per loop. It would be simple enough to add a version of epoll_ctl that took array arguments, but the existing epoll_ctl makes a lot of sense for the most common use cases. You either know all the descriptors ahead of time or you don't. If you do then you can add them all on startup and never touch it again, and one-time costs usually aren't worth optimizing. If you don't then you're usually modifying them one at a time anyway. And if the syscall takes them one at a time then you don't have to worry about maintaining the state yourself or memory management based on how many descriptors you have or anything like that.
asveikau 10y ago
It sounds like you can avoid the bug except you know, the part where you can't avoid bugs. Example: Let's say your program is multi-threaded and you have a bug where you close(2) a dangling file descriptor, which normally harmlessly returns EBADF but this time its integer value happens due to unlucky timing to get re-used for a brand-new socket which is registered with epoll... Then suddenly perhaps this becomes relevant in your program, even though any time you intentionally close the fd you de-register from epoll first. In other words, when debugging a program using epoll, I think it's still a relevant question. These corner-case type things where it seems like "obviously" you're never going to screw up are often where the most interesting bugs can surface.
- quotemstr 10y ago
  If you ever see EBADF, abort the program. You've screwed up something badly somewhere. Ignoring EBADF is like ignoring a double free: even if you can get away with it in the short term, it's indicative of something horrible down the road.
  - asveikau 10y ago
    Yes, for the reasons I already illustrated in my example. To be clear I am not advocating EBADF to be something you should live with. I am describing it as unintentional and as a bug. It is a time bomb for harmful race conditions.
naasking 10y ago
> Interesting writeup, seems like this only applies to people writing epoll abstraction layers though. [...] So I'm not sure calling it "fundamentally broken" is necessarily accurate. If you can't write an abstraction around it, it sounds pretty broken.
mfukar 10y ago
The bug is only there because the epoll interface necessitates you write the code to avoid it. That, to me, is the definition of a fundamentally broken interface.

by jgh 10y ago

bluejekyll 10y ago

Error prone? It definitely creates situations where bugs crop up. The example of close() is actually quite important, and something which is perhaps unexpected until you hit that bug and spend hours or days tracking it down. Definitely has happened to me.

joosters 10y ago

Yeah, I think 'fundamentally broken' is going into clickbait territory - these are API calls that are successfully used by a huge number of programs. epoll() is the most recent of these multiplexer syscalls and it's been around for a decade or so. The underlying message is that there are corner cases that exist and should be thought about, but that is true of most syscalls. The epoll() call itself is a strange API. As the series of articles says, it was introduced to solve the problem of the kernel having to register and unregister every listed FD on every invocation. However, as a trade-off, your program now needs to make a syscall each time it wants to add/remove an FD. So where you once had a single poll() call per event loop, you now have to fire off extra syscalls for each single FD on each new connection. Compare this to the API of FreeBSD's kqueue - it is another solution to avoid the linear costs of select()/poll(), but instead of making a syscall for registering/unregistering each new FD, kevent() is called with a list of FDs that you want to change. So an event loop using kqueue consists of just one syscall per loop.

zrm 10y ago
> Compare this to the API of FreeBSD's kqueue - it is another solution to avoid the linear costs of select()/poll(), but instead of making a syscall for registering/unregistering each new FD, kevent() is called with a list of FDs that you want to change. So an event loop using kqueue consists of just one syscall per loop. It would be simple enough to add a version of epoll_ctl that took array arguments, but the existing epoll_ctl makes a lot of sense for the most common use cases. You either know all the descriptors ahead of time or you don't. If you do then you can add them all on startup and never touch it again, and one-time costs usually aren't worth optimizing. If you don't then you're usually modifying them one at a time anyway. And if the syscall takes them one at a time then you don't have to worry about maintaining the state yourself or memory management based on how many descriptors you have or anything like that.

asveikau 10y ago

It sounds like you can avoid the bug except you know, the part where you can't avoid bugs. Example: Let's say your program is multi-threaded and you have a bug where you close(2) a dangling file descriptor, which normally harmlessly returns EBADF but this time its integer value happens due to unlucky timing to get re-used for a brand-new socket which is registered with epoll... Then suddenly perhaps this becomes relevant in your program, even though any time you intentionally close the fd you de-register from epoll first. In other words, when debugging a program using epoll, I think it's still a relevant question. These corner-case type things where it seems like "obviously" you're never going to screw up are often where the most interesting bugs can surface.

quotemstr 10y ago
If you ever see EBADF, abort the program. You've screwed up something badly somewhere. Ignoring EBADF is like ignoring a double free: even if you can get away with it in the short term, it's indicative of something horrible down the road.
- asveikau 10y ago
  Yes, for the reasons I already illustrated in my example. To be clear I am not advocating EBADF to be something you should live with. I am describing it as unintentional and as a bug. It is a time bomb for harmful race conditions.

naasking 10y ago

> Interesting writeup, seems like this only applies to people writing epoll abstraction layers though. [...] So I'm not sure calling it "fundamentally broken" is necessarily accurate. If you can't write an abstraction around it, it sounds pretty broken.

mfukar 10y ago

The bug is only there because the epoll interface necessitates you write the code to avoid it. That, to me, is the definition of a fundamentally broken interface.