6 ms·
I/O Multiplexing (select vs. poll vs. epoll/kqueue)
- sureglymop 1y agoGood read but I wish it included io_uring as well.
- marginalia_nu 1y agoIt's probably hard to include io_uring in something like this, without the article turning into an article mostly about io_uring. It's a cool API that can be incredibly fast, but it also comes with a very long list of caveats.
- sureglymop 1y agoWhen I learned about epoll it was at first entirely from man pages, then by looking at source code of async runtimes like tokio and libuv. I only learned about io_uring a few years after that. So, just mentioning that it exists may be interesting for readers. Nothing in-depth.
- lynx97 1y agoThere is no mention of epoll in thsi other then the heading.
- lstodd 1y agoIt's because epoll === kqueue mostly. Besides kqueue grew from FreeBSD, not OSX. Such ignorance saddens me much more.
- mort96 1y agoI wish the UNIXes had gone together and standardized a modern alternative to poll, maybe as part of POSIX. It sucks that any time I want to listen to IO events, I have to choose between old, low performance, cross-platform APIs and the new, higher-performance but Linux-only epoll.
- ahartmetz 1y agoFor sure. Though every platform does have it own high-performance alternative, with only kqueue shared by some less popular ones.
- usrnm 1y agoAren't there enough wrapper libraries for all programming languages that take care of this under the hood? You don't have to rely on libc only
- mort96 1y agoSure, there are wrapper libraries. But then I'm met with the question: do I add some big heavy handed IO wrapper library, or ... do I just call poll
- Galanwe 1y agoI wouldn't count uv/ev/etc as "big heavy IO wrapper library".
- mort96 1y agoI would, especially when nothing else in the program uses it and you just introduce it for one small thing in place of calling poll(). It's over 40 000 loc, over 70 000 including tests.
- paulddraper 1y agoI certainly would
- ninjin 1y agoWhich is why there is libevent [1]? [1]: https://libevent.org https://libevent.org Unless I am mistaken, OpenBSD base even explicitly codes against the older libevent API internally and ships it with each release, despite at the very least supporting kqueue, and thus gains better portability for a number of their tools this way. Personally, I just go with Posix select for small programs where performance is not critical anyway.
- commandersaki 1y agoNice article, though a few spelling mistakes that I thought was to distinguish it from AI slop, only to realise this was written a few years before the AI/GPT craze.
- tarruda 1y agoI have implemented a simple asyncio compatible micro event loop library in python. The goal was to understand the underlying mechanisms behind python's async/await and to help coworkers understand how event loops work under the hoods. The end result is somewhat interesting, as unlike traditional event loop libraries, it doesn't use callbacks as the scheduling primitive: https://gist.github.com/tarruda/5b8c19779c8ff4e8100f0b37eb5981ea https://gist.github.com/tarruda/5b8c19779c8ff4e8100f0b37eb59...
- quibono 1y agoI'm assuming epoll is covered implicitly by the section on kqueue. Are there any differences between the two besides the name?
- toast0 1y agoepoll returns a single value for events, and kqueue returns a struct. typedef union epoll_data { void *ptr; int fd; uint32_t u32; uint64_t u64; } epoll_data_t; vs struct kevent { uintptr_t ident; /* identifier for this event */ short filter; /* filter for event */ u_short flags; /* action flags for kqueue */ u_int fflags; /* filter flag value */ int64_t data; /* filter data value */ void *udata; /* opaque user data identifier */ uint64_t ext[4]; /* extensions */ }; For read/write events, ident is the FD and data is the number of bytes that can be read or written.
- eqvinox 1y ago> epoll/kqueue are replacements for their deprecated counterparts poll and select. Neither poll nor select are deprecated. They're just not good fits for particular use patterns. But even select() is fine if you just need to watch 2 FDs in a CLI tool. In fact, due to its footguns, I'd highly advise against epoll (particularly edge triggering) unless you really need it.
- MomsAVoxell 1y agoselect() is great for embedded daemons and user space signals handling, and so on. Just don't try to solve the 10,000x problem with it, by putting it on the Internet. Or, if you do, build it out properly. Or use epoll or kqueue.
- oconnor663 1y agoselect() is at least kind of deprecated, in that its own man page says not to use it in new code.
- toast0 1y agoI don't see it in the man page? https://man.freebsd.org/cgi/man.cgi?select https://man.freebsd.org/cgi/man.cgi?select The man page also suggests how you might increase the FD limit if needed. I still use select for a small number of FDs where overhead isn't a real concern, and select is a good fit.
- hippo22 1y agohttps://man7.org/linux/man-pages/man2/select.2.html https://man7.org/linux/man-pages/man2/select.2.html
- buckle8017 1y agoIn anything new you should use poll not select. They're basically identical apis but poll doesn't have a hard limit and works with high number fds.
- Luker88 1y agoI have vague memories of OSX kqueue not supporting all the usecases that FreeBSD kqueue does from many years ago. Have they reached feature parity?
- nesarkvechnep 1y agoI doubt it because applications, using kqueue, written for OSX can’t easily be ported to FreeBSD. ghostty is one such app.
- loeg 1y agoGhostty uses Mach ports on OS X in addition to kqueue. Source is here: https://github.com/mitchellh/libxev/blob/main/src/backend/kqueue.zig https://github.com/mitchellh/libxev/blob/main/src/backend/kq...
- khaledh 1y agoNeeds "(2020)" in the title.
- thasso 1y agoThis part is bewildering to me: > Now, if you try to watch file descriptor 2000, select will loop over fds from 0 to 1999 and will read garbage. The bigger issue is when it tries to set results for a file descriptor past 1024 and tries to set that bit field in say readfds, writefds or errorfds field. At this point it will write something random on the stack eventually crashing the process and making it very hard to debug what happened since your stack is randomized. I'm not too literate on the Linux kernel code, but I checked, and it looks like the author is right [1]. It would have been so easy to introduce a size check on the array to make sure this can't happen. The man page reads like FD_SETSIZE differs between platforms. It states that FD_SETSIZE is 1024 in glibc, but no upper limit is imposed by the Linux kernel. My guess is that the Linux kernel doesn't want to assume a value of FD_SETSIZE so they leave it unbounded. It's hard to imagine how anyone came up with this thinking it's a good design. Maybe 1024 FDs was so much at the time when this was designed that nobody considered what would happen if this limit is reached? Or they were working on system where 1024 was the maximum number of FDs that a process can open? [1]: The core_sys_select function checks the nfds argument passed to select(2) and modifies the fd_set structures that were passed to the system call. The function ensures that n <= max_fds (as the author of the post stated), but it doesn't compare n to the size of the fd_set structures. The set_fd_set function, which modifies the user-side fd_set structures, calls right into __copy_to_user without additional bounds checks. This means page faults will be caught and return -EFAULT, but out-of-bounds accesses that corrupt the user stack are possible.
- ajross 1y agoYou (and the author) are misunderstanding. These are all userspace pointers. If the process passes the kernel a buffer and tells it to access it past the end, the kernel will happily do so. It applies all the standard memory protection rules, which means that if your pointer is unmapped or unwritable, the kernel will signal the error (as a SIGSEGV) just as if the process had touched the memory itself. It's no different that creating a 1024 byte buffer and telling read() to read 2048 bytes into it. To be fair there's an API bug here in that "fd_set" is a fixed-size thing for historical compatibility reasons, while the kernel accepts arbitrarily large buffers now. So code cutting and pasting from historical examples will have a essentially needless 1024 FD limit. Stated differently: the POSIX select() has a fixed limit of file descriptors, the linux implementation is extensible. But no one uses the latter feature (because at that scale poll and epoll are much better fits) and there's no formal API for it in the glibc headers.
- lukaslalinsky 1y agoThe trouble with I/O multiplexing in a language like C is that the callbacks and state machines get quite complex as you need more functionality. In C++ you can at least do closures, so it's easier to manage. I recently wanted to add networking to my Zig project and decided to do some yak shaving and implemented a fiber runtime with async I/O to avoid the callback complexity. https://github.com/lalinsky/zio https://github.com/lalinsky/zio
- nesarkvechnep 1y agoAs usual, no FreeBSD support.
- qudat 1y agoWow nice! How does this compare to libxev?
- jfadfwddas 1y agoI was curious as well and looks like this abstracts over libxev: https://github.com/lalinsky/zio/blob/main/build.zig#L7 https://github.com/lalinsky/zio/blob/main/build.zig#L7
- lukaslalinsky 1y agoIndeed, it's a translation of the callback-based libxev events to coroutines. I ended up temporarily forking libxev, to add support for vectored I/O and other small fixes, but all those changes will be upstreamed.
- jfadfwddas 1y agoGreat stuff. I will be using this if/when I go back to zigging :)
- spacechild1 1y agoIn C++20 you can use asio + coroutines. I find it pretty nice to work with.
- drewg123 1y ago> kqueue (on macOS) Wish they'd give some credit to FreeBSD, where it originated..
- kcexn 1y agopoll is the POSIX specified I/O multiplexer so it has the advantage of being portable. Windows even supports a version of poll called WSAPoll. If you must implement your own event loop and you want your application to be portable, poll is still a good place to begin. O(N) demultiplexing time in the pollfd array is also not as brutal as it seems on modern hardware. The pollfd structure itself is only 8 bytes wide, so you can comfortably pack thousands of them into the L1 cache. Copying all of the elements that have an active event into a new smaller array before processing them is going to be fast enough for most cases.