5 ms·

epoll: The API that powers the modern internet (2022)

noncoml 3y ago
Worth reading some critique about epoll as well: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... Not to negate all the goodies that has given us
- JackSlateur 3y ago
  He does not speak about SO_REUSEPORT (https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/) tldr: you create a bunch of processes, each of them creates its own socket (bound to the same local port) and processes requests. The kernel split incoming connections amongst the processes. The issue: "short" and "long" requests are managed in the same way, so one process could be overloaded by "long" requests. There is no feedback loop, as far as I know. Also, he does not speak about SCM_RIGHTS (https://man7.org/linux/man-pages/man7/unix.7.html https://man7.org/linux/man-pages/man7/unix.7.html), which may be replaced by the newer pidfd_getfd (https://man7.org/linux/man-pages/man2/pidfd_getfd.2.html https://man7.org/linux/man-pages/man2/pidfd_getfd.2.html) tldr: you can pass a file descriptor between processes. So an overloaded process could pass some work to another worker. Epoll is a great component, yet it will not give you billions of qps alone
  - gpderetta 3y ago
    > He does not speak about SO_REUSEPORT (https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/) They do. Look for "Correct Solution" further down.
deleted 3y ago
[deleted]
commandersaki 3y ago
Would someone be kind enough to explain how a C program is structured to do asynchronous work with epoll without the use of async/await? Do you need to keep custom structures for each network socket that maintains a snapshot of the state of how things are progressing?
- j16sdiz 3y ago
  > Do you need to keep custom structures for each network socket that maintains a snapshot of the state of how things are progressing? In C, yes.
- willvarfar 3y ago
  it was eons ago that I looked at this so I don't remember the technical details, but here's a c++ framework that used epoll https://github.com/williame/hellepoll https://github.com/williame/hellepoll
  - gpderetta 3y ago
    Back in the '90s, ACE was the library that introduced/popularized reactor/proactor to C++. These days there are many libraries available. ASIO is quite popular, seastar, POCO, etc.
- eqvinox 3y ago
  > without the use of async/await > Do you need to keep custom structures for each network socket that maintains a snapshot of the state of how things are progressing? Yup. Everything is a callback and you dispatch those from your event loop. You're managing/implementing your own closures by (manually/explicitly) throwing all necessary data into some data structure (essentially the "async" part). Then make appropriate callback registration calls and return from your code (equivalent to "await").
- latch 3y ago
  I have a somewhat popular HTTP server library for Zig [1]. It started off as a thread-per-connection (with an optional thread pool), but when it became apparent that async wasn't going to be added back into the language any time soon, I switched to using epoll/kqueue. Both APIs allow you to associate arbitrary data (a `void *` in kqueue, or a union of `int/uin32_t/uint65_/void *` in epoll) with the event that you're registering. So when you're notified of the event, you can access this data. In my case, it's a big Conn struct. It contains things like the # of requests on this connection (to enforce a configured max request per connection), a timestamp where it should timeout if there's no activity. The Conn is part of an intrusive linked list, so it has a next: *Conn and prev: *Conn. But, what you're probably most curious about, is that it has a Request.State. This has a static buffer ([]u8) that can grow as needed to hold all the received data up until that point (or if we're writing the data, then the buffered data that we have to write). It's important to have a max # of connections and a max request size so you can enforce an upper limit on the maximum memory the library might use. It acts as a state machine to track up to what point it's parsed the request. (since you don't want to have to re-parse the entire request as more bytes trickle in). It's all half-baked. I can do receiving/sending asynchronously, but the application handler is called synchronously, and if that, for example, calls PG, that's probably also synchronous (since there's no async PG library in Zig). Makes me feel that any modern language needs a cohensive (as in standard library, or de facto standard) concurrency story. [1] https://github.com/karlseguin/http.zig https://github.com/karlseguin/http.zig*
DeathArrow 3y ago
Any OS has its own way to do async I/O. Linux is not unique in that regard.
- crabbone 3y ago
  Linux is somewhat unique because they tried many times, and it never worked quite well, "epoll" is the penultimate attempt as of today, "io_uring" is the last one. Linux is also somewhat unique, because most of the Internet, basically, you could say all of the Internet with a bit of hand-waving runs on Linux. So, even if other OSes have asynchronous I/O it doesn't affect the Internet.
bluetomcat 3y ago
It's a fundamental Linux API for writing event loops of any kind. In addition to watching sockets, you also have eventfd (alternative to semaphores and condition variables), timerfd (timers managed by the kernel), signalfd (handle signals safely and synchronously), pidfd (handle child process events). They allow you to handle heterogenous events inside that same loop, reducing the usage of threads. For async disk IO, io_uring is the way to go.
- kqr 3y ago
  Do you know of a good resource for learning how to write event loops using these primitives? I feel like there are some extremely common and robust patterns there, but my day job mostly involves high-level abstractions like async/await so I never get to dig into how it really works.
  - supriyo-biswas 3y ago
    David Beazley’s Python Concurrency from the Ground Up[1] is a great introduction. [1] https://www.youtube.com/watch?v=MCs5OvhV9S4 https://www.youtube.com/watch?v=MCs5OvhV9S4
  - dns_snek 3y ago
    You might be interested in a pure Zig implementation of these primitives by Mitchell in his "libxev" event loop library: https://github.com/mitchellh/libxev https://github.com/mitchellh/libxev
  - yla92 3y ago
    I am not expert in these but I thought Tornado's ioloop was readable enough for me to learn more event loops. Mostly, it was being implemented in pure Python. https://github.com/tornadoweb/tornado/blob/branch4.5/tornado/ioloop.py#L72 https://github.com/tornadoweb/tornado/blob/branch4.5/tornado... (Had to be in 4.5 because the newer versions 5.x and 6.x, it's switched to Python's stdlib asyncio)
  - anonymousDan 3y ago
    Have a look at how memcached does it. It's a bit more interesting than most examples I've seen as insofar as I can remember it combines epoll with multithreading. Be aware that many of the tutorials and criticisms of epoll are a bit outdated as the API has evolved a little over time.
mewse 3y ago
Worth noting that there's an equivalent of epoll on most platforms. On Windows there's IOCP, on Mac and BSD-derivates there's kqueue, and Linux has epoll, but to a first approximation they all do basically the same thing; instead of giving you a full-sized array of "active or not" results that you have to iterate across to detect activity on each of your sockets (as you get from the standard berkeley sockets 'select' and 'poll' APIs), they only inform you about the sockets that are actually active, so you can spend less CPU time iterating through that big results array. I can say from personal experience that when you've got a single server process that's monitoring 25,000 sockets on a Pentium Pro 200mhz box, it makes a huge difference! I'm a little surprised (and maybe skeptical) that it'd make a noticeable difference for the much smaller number of sockets that your average web server would be using, but.. maybe?
- kqr 3y ago
  > to a first approximation they all do basically the same thing; instead of giving you an array you have to iterate across they inform you about the sockets that are actually active Well, that's half the story. The other half is that select/poll is stateless, meaning the application–kernel bridge is flooded with data about which events you are interested in, despite the fact that this set usually doesn't change much between calls. kqueue and the like are stateful instead: you tell the kernel which events you are interested in and then it remembers that.
  - mewse 3y ago
    Good point, yes!
  - abhishekjha 3y ago
    Wait, so which is better? Stateful or stateless? How do you decide? Very new to these APIs and their usages.
    - kqr 3y ago
      Stateless is simpler to implement and easier to scale across multiple nodes, but comes with additional overhead. When polling the kernel for sockets, the overhead was a bigger cost than implementation complexity and horizontal scaling. (Implementation complexity is still a problem – see the discussions regarding the quality of epoll; and horizontal scaling is just not something desktop kernels do.)
q2dg 3y ago
Apache's Event MPM uses epoll too, not just Nginx
kqr 3y ago
This discussion brought me over to the kqueue paper: https://people.freebsd.org/~jlemon/papers/kqueue.pdf https://people.freebsd.org/~jlemon/papers/kqueue.pdf It's an easy read and highly recommended! I think we can learn a lot by immersing ourselves in how good APIs are designed.
eqvinox 3y ago
I feel this should really mention the epoll pitfalls (cf. https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... ) - though "fundamentally broken" is excessive hyperbole. Also this specific article is kinda hazy on pointing out epoll's improvements over previous APIs. One half is that event subscriptions are registered/submitted for a period longer than one syscall (which the article kinda-sorta says.) But the other half is that you also have an opaque field (mostly used as pointer) that the kernel returns back to you on event notifications, which removes an extra level of lookup in userspace. And lastly, FreeBSD has in the meantime acquired epoll support, and Linux is moving to the next-yet-again replacement in io_uring (which calling an epoll/poll replacement is an insult, it does much more.)
- another2another 3y ago
  >and Linux is moving to the next-yet-again replacement in io_uring (which calling an epoll/poll replacement is an insult, it does much more.) Yes, at least this time they support true async disk IO, which epoll doesn't.
- o11c 3y ago
  Almost nobody actually uses epoll at that kind of scale though. Almost everybody is just fine with `accept` from just one thread. and `SO_REUSEPORT` is safe if you're using a fixed thread pool; it only fails if you use a variable-size thread pool or if you're using multiple processes and one dies (though frankly, if a process dies, you're going to lose something anyway). The "but what if I `fork` and `close`" problem is purely theoretical I'm pretty sure.
keepamovin 3y ago
epoll (and libuv) is why I love nodejs <3 :)
alenmilk 3y ago
I remember the sysadmins trying to figure out why sqiud was so slow and tracing the system calls. It was because they did not use kqueue/epoll. When he asked the maintainer to improve on this he said squid was fast enough. This was the start of Varnish where one of main ideas was to use modern os-calls to speed up things.
nadaviv 3y ago
While somewhat lengthy, I found this post to be a great resource for learning about async and epoll: https://ibraheem.ca/posts/too-many-web-servers/ https://ibraheem.ca/posts/too-many-web-servers/ It starts with a simple single-request-at-a-time HTTP server implemented in Rust, then progresses to examples with multi-threading, non-blocking, epoll-based multiplexing, futures and async/await, showing the limitations and advantages at each step. Previous discussion: https://news.ycombinator.com/item?id=37176960 https://news.ycombinator.com/item?id=37176960
escanda 3y ago
So what's the difference with io_uring?
- jabl 3y ago
  Epoll is a readyness notification API. You tell epoll which fd's you're interested in, then you run a poll operation which blocks until at least one of the fd's you're interested in is ready. Then you separately need to do the actual IO (which is essentially just a memcpy() between kernel and user space). In contrast io_uring is a completion notification API. You give it a buffer to read or write from/to, and it'll notify you when the IO has been completed. Io_uring is also more flexible, and can for example be used for disk IO which epoll can't.
  - escanda 3y ago
    Makes sense. Also makes for much more efficient data polling since you get the buffer completed but not a random fd on which to perform yet another syscall.
newpavlov 3y ago
epoll is a horrible API, which has grown from the even worse select and poll API. I believe its entrenchment, despite existence of APIs like IOCP, is one of the failings of Linux, which even has negatively influenced programming languages (I am looking at you Rust). Others have already posted the "fundamentally broken" article, but to add an anecdote, after closely working with io-uring, I simply can not go back to the horrors of epoll and I wish we, as an industry, will migrate from it as soon as possible.
- deleted 3y ago
  [deleted]
username3 3y ago
What does Erlang use?
- superdisk 3y ago
  https://github.com/erlang/otp/blob/adf0d2204215c12a03188f5d5a149bec329f17af/erts/emulator/sys/common/erl_poll.c#L1861 https://github.com/erlang/otp/blob/adf0d2204215c12a03188f5d5...

Worth reading some critique about epoll as well: https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... Not to negate all the goodies that has given us

JackSlateur 3y ago
He does not speak about SO_REUSEPORT (https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/) tldr: you create a bunch of processes, each of them creates its own socket (bound to the same local port) and processes requests. The kernel split incoming connections amongst the processes. The issue: "short" and "long" requests are managed in the same way, so one process could be overloaded by "long" requests. There is no feedback loop, as far as I know. Also, he does not speak about SCM_RIGHTS (https://man7.org/linux/man-pages/man7/unix.7.html https://man7.org/linux/man-pages/man7/unix.7.html), which may be replaced by the newer pidfd_getfd (https://man7.org/linux/man-pages/man2/pidfd_getfd.2.html https://man7.org/linux/man-pages/man2/pidfd_getfd.2.html) tldr: you can pass a file descriptor between processes. So an overloaded process could pass some work to another worker. Epoll is a great component, yet it will not give you billions of qps alone
- gpderetta 3y ago
  > He does not speak about SO_REUSEPORT (https://lwn.net/Articles/542629/ https://lwn.net/Articles/542629/) They do. Look for "Correct Solution" further down.

deleted 3y ago

[deleted]

commandersaki 3y ago

Would someone be kind enough to explain how a C program is structured to do asynchronous work with epoll without the use of async/await? Do you need to keep custom structures for each network socket that maintains a snapshot of the state of how things are progressing?

j16sdiz 3y ago
> Do you need to keep custom structures for each network socket that maintains a snapshot of the state of how things are progressing? In C, yes.
willvarfar 3y ago
it was eons ago that I looked at this so I don't remember the technical details, but here's a c++ framework that used epoll https://github.com/williame/hellepoll https://github.com/williame/hellepoll
- gpderetta 3y ago
  Back in the '90s, ACE was the library that introduced/popularized reactor/proactor to C++. These days there are many libraries available. ASIO is quite popular, seastar, POCO, etc.
eqvinox 3y ago
> without the use of async/await > Do you need to keep custom structures for each network socket that maintains a snapshot of the state of how things are progressing? Yup. Everything is a callback and you dispatch those from your event loop. You're managing/implementing your own closures by (manually/explicitly) throwing all necessary data into some data structure (essentially the "async" part). Then make appropriate callback registration calls and return from your code (equivalent to "await").
latch 3y ago
I have a somewhat popular HTTP server library for Zig [1]. It started off as a thread-per-connection (with an optional thread pool), but when it became apparent that async wasn't going to be added back into the language any time soon, I switched to using epoll/kqueue. Both APIs allow you to associate arbitrary data (a `void *` in kqueue, or a union of `int/uin32_t/uint65_/void *` in epoll) with the event that you're registering. So when you're notified of the event, you can access this data. In my case, it's a big Conn struct. It contains things like the # of requests on this connection (to enforce a configured max request per connection), a timestamp where it should timeout if there's no activity. The Conn is part of an intrusive linked list, so it has a next: *Conn and prev: *Conn. But, what you're probably most curious about, is that it has a Request.State. This has a static buffer ([]u8) that can grow as needed to hold all the received data up until that point (or if we're writing the data, then the buffered data that we have to write). It's important to have a max # of connections and a max request size so you can enforce an upper limit on the maximum memory the library might use. It acts as a state machine to track up to what point it's parsed the request. (since you don't want to have to re-parse the entire request as more bytes trickle in). It's all half-baked. I can do receiving/sending asynchronously, but the application handler is called synchronously, and if that, for example, calls PG, that's probably also synchronous (since there's no async PG library in Zig). Makes me feel that any modern language needs a cohensive (as in standard library, or de facto standard) concurrency story. [1] https://github.com/karlseguin/http.zig https://github.com/karlseguin/http.zig*

DeathArrow 3y ago

Any OS has its own way to do async I/O. Linux is not unique in that regard.

crabbone 3y ago
Linux is somewhat unique because they tried many times, and it never worked quite well, "epoll" is the penultimate attempt as of today, "io_uring" is the last one. Linux is also somewhat unique, because most of the Internet, basically, you could say all of the Internet with a bit of hand-waving runs on Linux. So, even if other OSes have asynchronous I/O it doesn't affect the Internet.

bluetomcat 3y ago

It's a fundamental Linux API for writing event loops of any kind. In addition to watching sockets, you also have eventfd (alternative to semaphores and condition variables), timerfd (timers managed by the kernel), signalfd (handle signals safely and synchronously), pidfd (handle child process events). They allow you to handle heterogenous events inside that same loop, reducing the usage of threads. For async disk IO, io_uring is the way to go.

kqr 3y ago
Do you know of a good resource for learning how to write event loops using these primitives? I feel like there are some extremely common and robust patterns there, but my day job mostly involves high-level abstractions like async/await so I never get to dig into how it really works.
- supriyo-biswas 3y ago
  David Beazley’s Python Concurrency from the Ground Up[1] is a great introduction. [1] https://www.youtube.com/watch?v=MCs5OvhV9S4 https://www.youtube.com/watch?v=MCs5OvhV9S4
- dns_snek 3y ago
  You might be interested in a pure Zig implementation of these primitives by Mitchell in his "libxev" event loop library: https://github.com/mitchellh/libxev https://github.com/mitchellh/libxev
- yla92 3y ago
  I am not expert in these but I thought Tornado's ioloop was readable enough for me to learn more event loops. Mostly, it was being implemented in pure Python. https://github.com/tornadoweb/tornado/blob/branch4.5/tornado/ioloop.py#L72 https://github.com/tornadoweb/tornado/blob/branch4.5/tornado... (Had to be in 4.5 because the newer versions 5.x and 6.x, it's switched to Python's stdlib asyncio)
- anonymousDan 3y ago
  Have a look at how memcached does it. It's a bit more interesting than most examples I've seen as insofar as I can remember it combines epoll with multithreading. Be aware that many of the tutorials and criticisms of epoll are a bit outdated as the API has evolved a little over time.

mewse 3y ago

Worth noting that there's an equivalent of epoll on most platforms. On Windows there's IOCP, on Mac and BSD-derivates there's kqueue, and Linux has epoll, but to a first approximation they all do basically the same thing; instead of giving you a full-sized array of "active or not" results that you have to iterate across to detect activity on each of your sockets (as you get from the standard berkeley sockets 'select' and 'poll' APIs), they only inform you about the sockets that are actually active, so you can spend less CPU time iterating through that big results array. I can say from personal experience that when you've got a single server process that's monitoring 25,000 sockets on a Pentium Pro 200mhz box, it makes a huge difference! I'm a little surprised (and maybe skeptical) that it'd make a noticeable difference for the much smaller number of sockets that your average web server would be using, but.. maybe?

kqr 3y ago
> to a first approximation they all do basically the same thing; instead of giving you an array you have to iterate across they inform you about the sockets that are actually active Well, that's half the story. The other half is that select/poll is stateless, meaning the application–kernel bridge is flooded with data about which events you are interested in, despite the fact that this set usually doesn't change much between calls. kqueue and the like are stateful instead: you tell the kernel which events you are interested in and then it remembers that.
- mewse 3y ago
  Good point, yes!
- abhishekjha 3y ago
  Wait, so which is better? Stateful or stateless? How do you decide? Very new to these APIs and their usages.
  - kqr 3y ago
    Stateless is simpler to implement and easier to scale across multiple nodes, but comes with additional overhead. When polling the kernel for sockets, the overhead was a bigger cost than implementation complexity and horizontal scaling. (Implementation complexity is still a problem – see the discussions regarding the quality of epoll; and horizontal scaling is just not something desktop kernels do.)

q2dg 3y ago

Apache's Event MPM uses epoll too, not just Nginx

kqr 3y ago

This discussion brought me over to the kqueue paper: https://people.freebsd.org/~jlemon/papers/kqueue.pdf https://people.freebsd.org/~jlemon/papers/kqueue.pdf It's an easy read and highly recommended! I think we can learn a lot by immersing ourselves in how good APIs are designed.

eqvinox 3y ago

I feel this should really mention the epoll pitfalls (cf. https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/ https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-... ) - though "fundamentally broken" is excessive hyperbole. Also this specific article is kinda hazy on pointing out epoll's improvements over previous APIs. One half is that event subscriptions are registered/submitted for a period longer than one syscall (which the article kinda-sorta says.) But the other half is that you also have an opaque field (mostly used as pointer) that the kernel returns back to you on event notifications, which removes an extra level of lookup in userspace. And lastly, FreeBSD has in the meantime acquired epoll support, and Linux is moving to the next-yet-again replacement in io_uring (which calling an epoll/poll replacement is an insult, it does much more.)

another2another 3y ago
>and Linux is moving to the next-yet-again replacement in io_uring (which calling an epoll/poll replacement is an insult, it does much more.) Yes, at least this time they support true async disk IO, which epoll doesn't.
o11c 3y ago
Almost nobody actually uses epoll at that kind of scale though. Almost everybody is just fine with `accept` from just one thread. and `SO_REUSEPORT` is safe if you're using a fixed thread pool; it only fails if you use a variable-size thread pool or if you're using multiple processes and one dies (though frankly, if a process dies, you're going to lose something anyway). The "but what if I `fork` and `close`" problem is purely theoretical I'm pretty sure.

keepamovin 3y ago

epoll (and libuv) is why I love nodejs <3 :)

alenmilk 3y ago

I remember the sysadmins trying to figure out why sqiud was so slow and tracing the system calls. It was because they did not use kqueue/epoll. When he asked the maintainer to improve on this he said squid was fast enough. This was the start of Varnish where one of main ideas was to use modern os-calls to speed up things.

nadaviv 3y ago

While somewhat lengthy, I found this post to be a great resource for learning about async and epoll: https://ibraheem.ca/posts/too-many-web-servers/ https://ibraheem.ca/posts/too-many-web-servers/ It starts with a simple single-request-at-a-time HTTP server implemented in Rust, then progresses to examples with multi-threading, non-blocking, epoll-based multiplexing, futures and async/await, showing the limitations and advantages at each step. Previous discussion: https://news.ycombinator.com/item?id=37176960 https://news.ycombinator.com/item?id=37176960

escanda 3y ago

So what's the difference with io_uring?

jabl 3y ago
Epoll is a readyness notification API. You tell epoll which fd's you're interested in, then you run a poll operation which blocks until at least one of the fd's you're interested in is ready. Then you separately need to do the actual IO (which is essentially just a memcpy() between kernel and user space). In contrast io_uring is a completion notification API. You give it a buffer to read or write from/to, and it'll notify you when the IO has been completed. Io_uring is also more flexible, and can for example be used for disk IO which epoll can't.
- escanda 3y ago
  Makes sense. Also makes for much more efficient data polling since you get the buffer completed but not a random fd on which to perform yet another syscall.

newpavlov 3y ago

epoll is a horrible API, which has grown from the even worse select and poll API. I believe its entrenchment, despite existence of APIs like IOCP, is one of the failings of Linux, which even has negatively influenced programming languages (I am looking at you Rust). Others have already posted the "fundamentally broken" article, but to add an anecdote, after closely working with io-uring, I simply can not go back to the horrors of epoll and I wish we, as an industry, will migrate from it as soon as possible.

deleted 3y ago
[deleted]

username3 3y ago

What does Erlang use?

superdisk 3y ago
https://github.com/erlang/otp/blob/adf0d2204215c12a03188f5d5a149bec329f17af/erts/emulator/sys/common/erl_poll.c#L1861 https://github.com/erlang/otp/blob/adf0d2204215c12a03188f5d5...