8 ms·
Linux: What Can You Epoll?
- rwmj 4y agoIt's weird that (according to this document) you can epoll Unix domain sockets but not sockets created by socketpair(2). I thought socketpair created essentially two pre-connected Unix domain sockets.
- ajross 4y agoIndeed, I am all but certain you can epoll on socketpairs. That sounds like a mistake in the article.
- kentonv 4y agoHmm, I don't think that's it says (unless they edited it since your post?). It mentions socketpair explicitly as something that is epoll-friendly, and which you can use to communicate with another thread, in the case where you must create a thread to perform some blocking task but still want to get completion notification in the main thread via epoll.
- graham_king_3 4y agoYou can and they do. The article says: There are four epoll-able ways for processes or threads to communicate: - UNIX socket pairs: man socketpair.
- guenthert 4y agoThanks for the reminder that there is no non-blocking i/o for files residing on block devices.
- yxhuvud 4y agoBut there is, io_uring.
- m00dy 4y agoio_uring, a magical keyword I used to use in job interviews...
- healthandsafety 4y agoCare to elaborate?
- kortilla 4y agoEveryone says it’s better on paper but you rarely get to actually use it in real code.
- guenthert 4y agoThat is async i/o afaiu and not classic Unix non-blocking i/o (O_NONBLOCK given to 2 open()).
- yxhuvud 4y agoSure. But why does the difference matter? It is not as if epoll is classic Unix either.
- guenthert 4y agoepoll might not be, but poll is (depending on how one would interpret 'classic'). Anyhow, I wrongly assumed the difference mattered in respect of whether one could use io_uring in combination with epoll(). It turns out, one can [1] or [2]. [1] https://stackoverflow.com/questions/70132802/waiting-for-epoll-and-io-uring-simultaneously https://stackoverflow.com/questions/70132802/waiting-for-epo... [2] https://unixism.net/loti/tutorial/register_eventfd.html https://unixism.net/loti/tutorial/register_eventfd.html
- yxhuvud 4y agoHaving done my own share of uring bindings I wish I had found work places that appreciated that.
- Matthias247 4y ago> On Linux write to a regular file never blocks. Writing to a file copies data from our user space buffer to the kernel buffer and returns immediately. At some later point in time the kernel will send it to the disk. A regular file is hence always ready for writing and epoll wouldn’t add anything. Is that true? If it would be, the amount of data the Kernel would need to buffer is unbounded. I assumed there is a limit on the amount of buffered and not yet committed data, and when that is cross the call would block until more data is flushed to disk. Which is kind of the same as happens for TCP sockets. The `write()` call there doesn't really send data to the peer, it just submits data to the kernels send buffer, from where it will be asynchronously transmitted. Edit: Actually I will answer my own question and say I know it will block. I had deployed IO heavy applications in the past with instrumented read/write calls for IO operations in a threadpool. Even though typical IO times are well below 1ms, under extremely high load latencies of more than 1s could be observed, which is far from "not blocking".
- kotlin2 4y agoThe write call returns how many bytes were accepted: https://man7.org/linux/man-pages/man2/write.2.html https://man7.org/linux/man-pages/man2/write.2.html > The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium, or the RLIMIT_FSIZE resource limit is encountered (see setrlimit(2)), or the call was interrupted by a signal handler after having written less than count bytes. (See also pipe(7).)
- wtallis 4y agoThat doesn't answer the question. Blocking isn't a matter of how much data is written, but a matter of when the system call completes. Other parts of that man page imply that write(2) may block, unless the fd was opened with O_NONBLOCK (in which case you'll get an EAGAIN error instead of it blocking).
- throwaway09223 4y ago> (in which case you'll get an EAGAIN error instead of it blocking). You won't. O_NONBLOCK cannot be used with regular files. That part of the manpage is discussing other non-socket file types. Disk i/o via write(2) is always a blocking call. Always. 100% of the time, no exceptions.
- bfrog 4y agowhy epoll at all, the new hotness is io_uring, fire away your iovecs, check back later
- rwmj 4y agoYou can go from select/poll to epoll relatively easily, but I've found that to use io_uring you have to substantially rearchitect your whole program (if you want any performance benefit). Actually I'd love to be wrong about this, but I've not found a way to easily retrofit io_uring into programs/libraries that are already using either synchronous operations or poll(2).
- jasonzemos 4y agoio_uring is basically a drop-in for epoll. It has an intrinsic performance benefit because multiple operations can be both submitted and completed in a single action. Rearchitecting is only optional when going further by replacing standalone syscalls with io_uring operations. In the case of poll(2) I believe it should be no more difficult than refactoring for epoll.
- wahern 4y agoWith io_uring, every line in an application that calls read/recv needs to be refactored, along with much of the surrounding context. io_uring doesn't replace poll/epoll, it effectively replaces typical event loop frameworks. You can integrate io_uring into pre-existing event loop frameworks, but the event loop framework will end up as a 99% superfluous wrapper, at least on Linux. Note that many applications don't use event loop frameworks. For simple applications they can be overkill. Even for more complex applications, it may be cleaner to use restartable semantics (i.e. same semantics as read--just call me again), especially for libraries or components that want to be event loop agnostic.
- jasonzemos 4y ago> io_uring doesn't replace poll/epoll, it effectively replaces typical event loop frameworks. I'm sorry but that statement is incorrect. The same functionality is accomplished by either io_uring or epoll. > every line in an application that calls read/recv needs to be refactored If the GP finds it easy to refactor from poll to epoll they will find it no different to refactor from poll to io_uring; the only caveat being they won't exceed epoll's performance with the io_uring until the standalone syscalls are further refactored. They don't need to be. The statement is incorrect.
- tlsalmin 4y agoJust skimmed through the article, since I'm just here to testify that the most important revelation for me on writing APIs was that you can put and epoll_fd in an epoll_fd. This allows the API to have e.g. a single epoll_fd that contains all outbound connections, timers, signalfds and inotifys mentioned in the articled. Then the e.g. daemon using the APIs can have an epoll_fd per library it is using and just be sitting in the epoll_wait loop ready to fire library_x_process() call when events arrive.
- kentonv 4y agoAnother use case for this: Say you have a set of "jobs" each composed of many "tasks" (each waiting for some event). The "jobs" are able to run concurrently on different threads, but the "tasks" must not run concurrently with other tasks in the same job because they might share data structures without synchronization. (This is a pretty common pattern in a lot of big servers.) Now you want to make sure you utilize multiple cores effectively. The naive approaches are: 1. Create a thread per job, each waiting on its own epoll specific to the job. This may be expensive if there are many jobs, and could allow too much concurrency. 2. Have a single epoll and a pool of threads waiting on it. Each thread must lock a mutex for the job that owns the task it's going to run. But a thread could receive an event for a task belonging to a job that's already running on another thread, in which case it has to synchronize with that other thread somehow, which is a pain. Be careful not to create a situation where all threads are blocked on the mutex for one job while other jobs are starved. Epoll nesting presents a clean solution: 3. Create an epoll per job, plus an outer epoll that waits on other epolls. A pool of threads waits on the outer epoll, which signals when a per-job epoll becomes ready. The thread receiving that event then takes ownership of the per-job epoll until the event queue is empty.
- remram 4y ago> The thread receiving that event then takes ownership of the per-job epoll until the event queue is empty. What do you mean by that? Take that per-job epoll out of the outer epoll?
- 4y ago
- kentonv 4y agoI highly recommend that you do NOT use signalfd to get notification of signals through epoll. Instead, block (mask) the signal, set a signal handler, and use epoll_pwait() to atomically unblock it while you wait for events. Note that in this setup, your signal handler callback need not be async-signal-safe, since you know the precise state of the calling thread: it's invoking epoll_pwait(). This sidesteps most of the pain of using signals which might otherwise make you think you want signalfd. Two reasons not to use signalfd: 1. signalfd has weird semantics that don't match what you'd normally expect from a file descriptor. When you read from a signalfd, it tells you signals queued on the thread that called read(), NOT the thread that created the signalfd. Worse, if you add signalfd to an epoll, the epoll will report readiness based on the thread that used epoll_ctl() to add the signalfd, which may be different from the thread that is reading from the epoll. So you might get a notification that the signalfd is ready, but then read the signalfd and find there are no signals, and then wait on the epoll again just to have it tell you again that this signalfd is ready. 2. It turns out that signalfd's implementation has some severe lock contention issues. I learned this through my own experimentation recently. In my experiment, I had 5000 threads each waiting on an epoll that included a signalfd. When delivering a thread-specific signal to each of the 5000 threads at once, the process spent 2+ MINUTES of CPU time spinning on spinlocks in the kernel before completing all the event deliveries. The time spent was O(n^2) to the number of threads. When I switched to an epoll_pwait()-based implementation, the same task took a few milliseconds. Here's the PR where I switched KJ's event loop (used in Cap'n Proto and Cloudflare Workers) to use epoll_pwait(): https://github.com/capnproto/capnproto/pull/1511 https://github.com/capnproto/capnproto/pull/1511
- kelnos 4y agoThe big downside of using a traditional signal handler is that the only way to get your own data into the handler function is through global variables (or thread locals). While you can certainly make an exception just for that one thing, it feels gross to do so. And you can also just defer processing to your main loop by setting a flag or writing to a pipe, but those things still need to be global variables. I didn't know about signalfd's limitations before reading your post, and was happy that signalfd could eliminate the need for global variables when doing signal handling. Shame that's not really the case.
- sylware 4y agoI wrote many of my own programs on elf/linux: I do epoll as much as I can. The only troubling thingy is the lack of classification of signals, those who are synchronous by nature and the other ones. For instance, in a monothreaded application, segfault won't be delivered via epoll... At the same time, it is still important to keep the asynchronous API for signals that for lower latency, but then, only the realtime behaviour should be kept since this is really where latency does matter.
- cryptonector 4y agoSynchronous signals can't possibly be delivered via an asynchronous mechanism like epoll. Well, I suppose they could, but only if the thread that tripped the synchronous signal is put to sleep until some other thread runs a handler for that signal, but this requires APIs that (to my knowledge, except... ptrace) don't exist, and there's no need for them. If you can handle a SIGBUS or SIGSEGV, you don't need to handle them asynchronously, but on the other hand if you can't handle them, then the process will die, so there's just no need to handle these asynchronously.
- sylware 4y agoExactly what I said: this needs clarification. Maybe adding explicitely the 2 flavors of signals.
- cryptonector 4y agoNo, it doesn't need clarification. Synchronous signals are always always synchronous, therefore it's obvious that they cannot be handled asynchronously with <fill-in-the-blank-async-system>. It's really clear. And if you stop to think about why synchronous signals are synchronous, it's even more clear.
- sylware 4y agoI disagree, I think it is not clear enough in POSIX.
- karthikmurkonda 4y agoYep
- emilfihlman 4y agoRegular files not having a non blocking mode is one of the biggest and gravest idiocies on linux land. And there's one even worse: even having the concept of uninterruptiple sleep (D).
- bitwize 4y agoWhy epoll when you can io_uring? In Rust?
- kocial 4y agoYour link to `Go` project has incorrect link at footer "in practice" section.
- graham_king_3 4y agoFixed, thank you.