3 ms·

It's a fundamental Linux API for writing event loops of any kind. In addition to watching sockets, you also have eventfd (alternative to semaphores and conditio

It's a fundamental Linux API for writing event loops of any kind. In addition to watching sockets, you also have eventfd (alternative to semaphores and condition variables), timerfd (timers managed by the kernel), signalfd (handle signals safely and synchronously), pidfd (handle child process events). They allow you to handle heterogenous events inside that same loop, reducing the usage of threads. For async disk IO, io_uring is the way to go.

kqr 3y ago
Do you know of a good resource for learning how to write event loops using these primitives? I feel like there are some extremely common and robust patterns there, but my day job mostly involves high-level abstractions like async/await so I never get to dig into how it really works.
- supriyo-biswas 3y ago
  David Beazley’s Python Concurrency from the Ground Up[1] is a great introduction. [1] https://www.youtube.com/watch?v=MCs5OvhV9S4 https://www.youtube.com/watch?v=MCs5OvhV9S4
- dns_snek 3y ago
  You might be interested in a pure Zig implementation of these primitives by Mitchell in his "libxev" event loop library: https://github.com/mitchellh/libxev https://github.com/mitchellh/libxev
- yla92 3y ago
  I am not expert in these but I thought Tornado's ioloop was readable enough for me to learn more event loops. Mostly, it was being implemented in pure Python. https://github.com/tornadoweb/tornado/blob/branch4.5/tornado/ioloop.py#L72 https://github.com/tornadoweb/tornado/blob/branch4.5/tornado... (Had to be in 4.5 because the newer versions 5.x and 6.x, it's switched to Python's stdlib asyncio)
- anonymousDan 3y ago
  Have a look at how memcached does it. It's a bit more interesting than most examples I've seen as insofar as I can remember it combines epoll with multithreading. Be aware that many of the tutorials and criticisms of epoll are a bit outdated as the API has evolved a little over time.
the8472 3y ago
io_uring and epoll can be combined, in both directions. Either add poll SQEs to the ring for an epollfd or have the uring submit readiness events to an eventfd that can be added to epoll.
ghusbands 3y ago
Of course, io_uring has had many security vulnerabilities, as it essentially forms a new ad-hoc programming language with its sequencing operations and doesn't seem to have a cohesive story around security. It will likely suffer many more vulnerabilities. This is presumably why many corporations, including Google, limit its use in production. As a lot of complication and issues seem to be around sequencing and the adhoc language, hopefully the community will eventually settle on using eBPF for all programming within io_uring [1]. [1] https://lwn.net/Articles/847951/ https://lwn.net/Articles/847951/

kqr 3y ago

Do you know of a good resource for learning how to write event loops using these primitives? I feel like there are some extremely common and robust patterns there, but my day job mostly involves high-level abstractions like async/await so I never get to dig into how it really works.

supriyo-biswas 3y ago
David Beazley’s Python Concurrency from the Ground Up[1] is a great introduction. [1] https://www.youtube.com/watch?v=MCs5OvhV9S4 https://www.youtube.com/watch?v=MCs5OvhV9S4
dns_snek 3y ago
You might be interested in a pure Zig implementation of these primitives by Mitchell in his "libxev" event loop library: https://github.com/mitchellh/libxev https://github.com/mitchellh/libxev
yla92 3y ago
I am not expert in these but I thought Tornado's ioloop was readable enough for me to learn more event loops. Mostly, it was being implemented in pure Python. https://github.com/tornadoweb/tornado/blob/branch4.5/tornado/ioloop.py#L72 https://github.com/tornadoweb/tornado/blob/branch4.5/tornado... (Had to be in 4.5 because the newer versions 5.x and 6.x, it's switched to Python's stdlib asyncio)
anonymousDan 3y ago
Have a look at how memcached does it. It's a bit more interesting than most examples I've seen as insofar as I can remember it combines epoll with multithreading. Be aware that many of the tutorials and criticisms of epoll are a bit outdated as the API has evolved a little over time.

the8472 3y ago

io_uring and epoll can be combined, in both directions. Either add poll SQEs to the ring for an epollfd or have the uring submit readiness events to an eventfd that can be added to epoll.

ghusbands 3y ago

Of course, io_uring has had many security vulnerabilities, as it essentially forms a new ad-hoc programming language with its sequencing operations and doesn't seem to have a cohesive story around security. It will likely suffer many more vulnerabilities. This is presumably why many corporations, including Google, limit its use in production. As a lot of complication and issues seem to be around sequencing and the adhoc language, hopefully the community will eventually settle on using eBPF for all programming within io_uring [1]. [1] https://lwn.net/Articles/847951/ https://lwn.net/Articles/847951/