9 ms·
File Descriptor Limits
- josephcsible 6y agoThis is definitely a Lennart Poettering post: it advocates for a change that will break existing code, and then says that said code "should probably be considered just buggy".
- xyzzy_plugh 6y agoHe's arguing that it already is buggy, and I agree. The fact that select doesn't work on more than 1024 fds is terrible. It's incredibly easy to break things by changing the limits. Bumping FD limits is extremely common. I'm honestly surprised I've never run into this first hand, but perhaps I wouldn't have noticed even if I had. Sure, Lennart is perhaps the Antichrist of Linux, but I'm pretty positive he's the Antichrist we need. The long-lived fear of breaking existing code is certainly valid, but at the same time, why not try and see what will break before we write something off. It's often not much. It's practically the only way for us to shed the crufty bits, and boy are there a lot of those.
- phone8675309 6y agoLennart can't be the Antichrist of Linux because the antichrist is beloved by all. :-)
- moises_silva 6y ago> The fact that select doesn't work on more than 1024 fds is terrible. Pedantic clarification, but it's worse than that. It doesn't work on any fd with a value higher than FD_SETSIZE (default is 1024) - 1. You could be working with single fd with a value of 1024 and it would not work. On the bright side you can extend this limit pretty easily, but it's certainly a good reason to avoid select whenever possible.
- phendrenad2 6y agoTo be clear - he's not advocating for a change to Linux or systemd. He's reiterating an already commonly accepted best practice going forward. ...which unfortunately due to the Linux ecosystem, where people apparently like to inject code into running processes (https://skarnet.org/software/nsss/nsswitch.html https://skarnet.org/software/nsss/nsswitch.html) (can't imagine this is brittle at all), can cause your program to break if someone injects a module that doesn't ALSO follow this best practice into your program.
- tadfisher 6y agoSee also: PAM, probably a much more enticing target. I wonder if something like the eBPF compiler could be repurposed here, to generate lightweight "binaries" for a configuration that can be linked without needing to dynamically link their own dependencies at runtime.
- LukeShu 6y agoFew programs load PAM modules, essentially all programs load NSS modules.
- dale_glass 6y agoI really hope Poettering decides to fix PAM too. It's well overdue. IMO PAM should be replaced with a daemon. This way all the code that deals with authentication can be isolated into its own process, and my screensaver doesn't need root access to check my password against /etc/shadow. It'll also simplify SELinux settings.
- jvreeland 6y agoI mean we've known for a while select is broken that code is probably buggy and insecure. https://sourceware.org/bugzilla/show_bug.cgi?id=10352#c1 https://sourceware.org/bugzilla/show_bug.cgi?id=10352#c1 https://marc.info/?l=bugtraq&m=110660879328901 https://marc.info/?l=bugtraq&m=110660879328901
- toast0 6y agoWhat's changing here that breaks existing code? I'm not a Linux person, but select requiring help to work beyond FD 1023 isn't new. It also happens on FreeBSD and MacOS, so either you recompile your software (including glibc on linux, apparently) with a bigger FD_SETSIZE, or you use one of the better than select interfaces that doesn't care. If you tuned your FD limit higher, you're already on the hook for knowing about it. Keeping the default soft limit of 1024 means that programs that don't fiddle with the soft limit can have a working select, and programs that do fiddle with the soft limit should know better. Auto tuning the hard limit to something appropriately sized seems reasonable, but fiddling with the soft limit may end up with this select problem, so it's nice that they didn't fiddle with it. Also, I'm deeply surprised to be defending Mr. Poettering, but in this case, he's right.
- tomcam 6y agoAh, relief. Helped me hugely to understand that most resources should actually be thought of file descriptors as opposed to files. I took the popular literature too literally.
- pengaru 6y agoNo new code should be using select(). Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE. I've personally had to debug and fix crashes due to this assumption at multiple jobs. Back at Hostway decades ago we had Apache+SSL instances randomly crashing which turned out to be due to something opening /dev/random and using select() on the returned file descriptor in a .so. This only happened when the machine had enough virtual hosts to exceed FD_SETSIZE on the spurious open of /dev/random. At Bizanga we had random crashes in their "carrier grade mail server" software, which would link in proprietary modules from vendors like Symantec for realtime virus scanning before even accepting messages for delivery. This stuff ran in-process for performance reasons, just in separate threads. Being "carrier grade" meant thousands of concurrent connections -> thousands of open files. Those vendor-provided plugins would phone home behind the scenes for virus db updates. Turned out they were using select() on the socket connected for the update, if there were enough active sessions at the time, the socket's value would exceed FD_SETSIZE and boom: segfault. People tend to forget that it's the value of the fds you pass to select() which can't exceed FD_SETSIZE, not just the quantity of file descriptors. It could be just one file descriptor, but if it's a high enough value exceeding FD_SETSIZE, it will segfault. The value is used as an index into a fixed size array. I ended up making an execution wrapper @ Bizanga to launch everything with >FD_SETSIZE file descriptors already opened once I saw the vendor modules were linking select(). We found numerous crashes this way, without waiting for them to occur in production.
- ridiculous_fish 6y agoIs there a portable alternative to pselect()? There's ppoll(), but it's missing on some platforms.
- wahern 6y agoAt this point I believe ppoll has been added to every mainstream Unix except macOS. It's also been accepted for the next POSIX revision. For many years pselect on macOS didn't actually fix the race condition, notwithstanding its UNIX03 certification. pselect was just a naive wrapper around select that didn't atomically unmask signals. So I ended up having to create a kqueue-based implementation for use on macOS and stragglers like OpenBSD which at that time lacked pselect entirely. See https://github.com/wahern/cqueues/blob/87f14f9/src/cqueues.c#L2489 https://github.com/wahern/cqueues/blob/87f14f9/src/cqueues.c... I can't remember if pselect has been fixed on macOS. The same approach could be used to emulate ppoll on macOS. Though epoll, kqueue, and Solaris ports are so similar that you can easily wrap all of them behind a thin shim. See https://github.com/wahern/cqueues/blob/87f14f9/src/lib/kpoll.c https://github.com/wahern/cqueues/blob/87f14f9/src/lib/kpoll... (Caution: it's an incomplete refactor of a similar shim I used in some non-open source code. The original code is even simpler and has seen extensive use. The refactored shim should work but is untested.) AIX is really the only mainstream Unix lacking an API w/ equivalent semantics. AIX has a better-than-poll interface, but it doesn't work anything like the others. Notably, epoll, kqueue, and Solaris ports control descriptors are all pollable in their own right, permitting you to nest different [readiness polling-based] event loop frameworks.
- lathiat 6y agoThis issue has been driving me crazy and I spent a while a couple of times trying to figure out why the limit was 1024 and if we could bump the default. I got sidetracked and never finished it either time. I also separately knew about the select() issue but hadn't linked the two. This sounds like a great change in that at least software can opt-in without having to even edit their systemd unit.
- welder 6y agoAll parent processes above you need to first increase their ulimit. Check the blog link in my other post here for details. Changing the daemon's ulimit in with systemd is actually super easy with one line: LimitNOFILE=<your-limit>
- lathiat 6y agoYep I know about that, but was more specifically interested in the case of not having to make such changes and having it just work (tm). Resource-wise having a 50,000 or 500,000 default ulimit wouldn't matter. But we find out here, it matters for compatability. But with the systemd change he mentioned (which for Ubuntu, exists in 20.04 but not 18.04) - that the HARD limit is now 512k - applications can chose to adjust the ulimit from inside the application that is not running as root. So all modern software can just, if it knows its OK, make the call during startup to increase the limit to whatever it thinks is needed. Then we dont have to override.conf systemd services every time you hit a scaling limit or think about it.
- quotemstr 6y agoAgreed with Poettering here. The FD limit is a terrible anachronism.
- deleted 6y ago[deleted]
- Sebb767 6y agoThis might be a stupid question, but why couldn't select be extended to accept fd's >=1024? I don't see "not segfaulting" breaking existing behavior and it would solve basically all issues with legacy programs. Yes, it might lead to select being used a bit longer than it otherwise would be, but that would probably be less of a pain than fighting segfaults caused by a `select()` somewhere in a linked library.
- wahern 6y agoselect(2) does support fds >= 1024. At least, it does on Linux and many other systems. The issue is that <sys/select.h> defines the type fd_set, a struct containing a fixed-size bit field as an array of integers. The maximum descriptor number (i.e. the bit field size) is reported by the constant FD_SETSIZE. Thus, conceptually fd_set is defined like typedef struct { int fds_bits[howmany(FD_SETSIZE, sizeof (int) * CHAR_BIT)] } fd_set; <sys/select.h> also provides a set of macros for initializing the structure (FD_ZERO), and testing (FD_ISSET) and setting (FD_SET, FD_CLR) descriptor numbers. But these macros have no way of reporting if a specified descriptor overflows the bit array. In all the implementations I'm familiar with, including Linux/glibc and Linux/musl, they don't even check, and will happily overwrite the array if given a descriptor greater than or equal to FD_SETSIZE. But the maximum descriptor number in the bit field is reported to the kernel as an argument to select. And most kernels, including Linux, obey this parameter[1], and thus impose no intrinsic limit. If userspace provides a validly allocated and initialized buffer all will work just fine. But how to allocate and use a properly sized buffer? Some systems, like OpenBSD, allow you to define FD_SETSIZE (or a similar constant) before including <sys/select.h>. On other systems, like modern Linux/glibc or Linux/musl, you have to get clever, like doing something like #define MYFD_SETSIZE 100000 #define MYFD_SETSIZE_OVERFLOW (MYFD_SETSIZE - FD_SETSIZE) union { fd_set fdset; char extra[sizeof (fd_set) + howmany(MYFD_SETSIZE_OVERFLOW, CHAR_BIT)]; } u; but then be careful not to rely on FD_ZERO, which won't know to zero out the high bits. [1] The same is true of (struct sockaddr_un).sun_path. On most systems you can pass Unix domain socket paths longer than 108 or whatever the local constant is, and the kernel will happily use the full path, though there may be another internal limit, like 256 or 1024. It can do this because syscalls like bind(2) and connect(2) take a separate addrlen parameter. One possibly shocking consequence to many people is that types like `struct sockaddr_storage` are not sufficient to represent every possible socket address. And unless you check the output of the addrlen pointer parameter to syscalls like accept(2), you may have received a truncated address without knowing it.
- strken 6y agoI have no idea how a man page for a Linux syscall gets defined or updated. What would it take to make the man page clearer on how bad select is, as per the article?
- welder 6y ago> And unfortunately RLIMIT_NOFILE is inherited down the process tree unless explicitly set. I go into common ways to configure the NOFILE/ulimit here: https://wakatime.com/blog/47-maximize-your-concurrent-web-server-connections https://wakatime.com/blog/47-maximize-your-concurrent-web-se... Mainly, keep checking the ulimit of your daemon process while configuring each layer until the ulimit bubbles down and shows up on the process.
- friend-monoid 6y agoPoettering makes some really valid points in this post; you shouldnt be using select. And the resource limits are weird. But I just want to consider that open fds are a resource hog, and you should also strive to close fds sooner rather than later. Most applications will stay well below 1024 fds, and if they consume more then that - it’s probably a bug. Servers, sure; each connection can easily consume 2-3 fds, that’s 300-500 simultaneous connections - that’s nothing. Systemd and other service supervisors, sure, with pidfds, timerfds, sockets, proc file sockets and whatever else. But for your average image editing softwares, email clients, CLIs, etc; id think it’s probably better to keep the limit? That way you are saving yourself from badly written software draining too much resources.
- majke 6y agoA quick plug: A brief history of select() syscall https://idea.popcount.org/2016-11-01-a-brief-history-of-select2/ https://idea.popcount.org/2016-11-01-a-brief-history-of-sele... (for example: did you know early BSD had a limit of 20(!) file descriptors?)
- apaprocki 6y agoMy favorite fd limit isn’t this one — it’s the default limit of 256 on Solaris for fopen(), etc. Unless you compile differently you only get 256 in a 32-bit app. Backwards compatibility to the grave...