6 ms·

If you think select isn't broken, show me a small server written using it that can handle more than 1024 concurrent clients.

pcwalton 10y ago
> If you think select isn't broken, show me a small server written using it that can handle more than 1024 concurrent clients. Divide your FDs into groups of 1024 each. Put each group in a different worker thread, each connected to the main thread with a Unix socket. Select over the group in each worker thread; when a socket wakes up, notify the main thread. Select over the worker connections in the main thread. You laugh, but it's a documented approach on Windows [1]. [1]: https://msdn.microsoft.com/en-us/library/windows/desktop/ms687025(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
- qwertyuiop924 10y ago
  That actually doesn't sound like the worst solution in the world. Not really a good one, either, but not as awful as you make it out to be. But I don't touch the kernel, so I could be wrong there.
zzzcpan 10y ago
This is an artificial limit. On many systems it can handle way more FDs than that if you use it directly, bypassing wrappers. Libev does that I believe, so any server on top of libev can use unlimited select.
- justinsaccount 10y ago
  Well, using libev to work around limitations in select is not really a simple server using select.. and if you're using libev wouldn't you not ideally be using select in the first place?
  - zzzcpan 10y ago
    It's not like that. As a system call select doesn't really have those limitations. Choosing an interface and wrappers to work with it is up to the programmer, be it libc or libev. As for ideally, using select as a default is a good idea for as long as you can get away with, everything else has too many caveats and working around them is tricky as you must relearn everything libev and nginx learned over the years if you haven't been paying close attention to the problems they encountered.
    - justinsaccount 10y ago
      Intersting. So it's just libc that has the broken FD_SETSIZE stuff?
anarazel 10y ago
Something not being usable for a specific usecase doesn't mean $thing is broken. It means it's not applicable for that usecase. My bike is broken, it doesn't go 550mph!!1 Edit: adapted stupid example to be differently stupid.
- pekk 10y ago
  This kind of broken reasoning is common in certain communities. Example: this programming language is broken, it can't be used to write drivers for the Linux kernel!
  - justinsaccount 10y ago
    No, it's more like "This feature is broken, if you attempt to use it for its intended purpose it will fail horribly once you try running it at scale".
- justinsaccount 10y ago
  Really? multiplexing file descriptors IS the specific use case of select. It would be more like "My bike is broken, when I try to ride more than 12 blocks the wheels fall off". Also, it's not even that select can't handle more than 1024 fds, it's that it can't handle a file descriptor above 1024 at all, so if your app opens a bunch of data files before accepting connection, the limit could be as low as 10 concurrent connections before select fails.
sroussey 10y ago
This is a problem today with PHP unfortunately, particularly when compiled into Apache.

pcwalton 10y ago

> If you think select isn't broken, show me a small server written using it that can handle more than 1024 concurrent clients. Divide your FDs into groups of 1024 each. Put each group in a different worker thread, each connected to the main thread with a Unix socket. Select over the group in each worker thread; when a socket wakes up, notify the main thread. Select over the worker connections in the main thread. You laugh, but it's a documented approach on Windows [1]. [1]: https://msdn.microsoft.com/en-us/library/windows/desktop/ms687025(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...

qwertyuiop924 10y ago
That actually doesn't sound like the worst solution in the world. Not really a good one, either, but not as awful as you make it out to be. But I don't touch the kernel, so I could be wrong there.

zzzcpan 10y ago

This is an artificial limit. On many systems it can handle way more FDs than that if you use it directly, bypassing wrappers. Libev does that I believe, so any server on top of libev can use unlimited select.

justinsaccount 10y ago
Well, using libev to work around limitations in select is not really a simple server using select.. and if you're using libev wouldn't you not ideally be using select in the first place?
- zzzcpan 10y ago
  It's not like that. As a system call select doesn't really have those limitations. Choosing an interface and wrappers to work with it is up to the programmer, be it libc or libev. As for ideally, using select as a default is a good idea for as long as you can get away with, everything else has too many caveats and working around them is tricky as you must relearn everything libev and nginx learned over the years if you haven't been paying close attention to the problems they encountered.
  - justinsaccount 10y ago
    Intersting. So it's just libc that has the broken FD_SETSIZE stuff?

anarazel 10y ago

Something not being usable for a specific usecase doesn't mean $thing is broken. It means it's not applicable for that usecase. My bike is broken, it doesn't go 550mph!!1 Edit: adapted stupid example to be differently stupid.

pekk 10y ago
This kind of broken reasoning is common in certain communities. Example: this programming language is broken, it can't be used to write drivers for the Linux kernel!
- justinsaccount 10y ago
  No, it's more like "This feature is broken, if you attempt to use it for its intended purpose it will fail horribly once you try running it at scale".
justinsaccount 10y ago
Really? multiplexing file descriptors IS the specific use case of select. It would be more like "My bike is broken, when I try to ride more than 12 blocks the wheels fall off". Also, it's not even that select can't handle more than 1024 fds, it's that it can't handle a file descriptor above 1024 at all, so if your app opens a bunch of data files before accepting connection, the limit could be as low as 10 concurrent connections before select fails.

sroussey 10y ago

This is a problem today with PHP unfortunately, particularly when compiled into Apache.