29 ms·

You probably shouldn't share epoll FDs across threads for performance reasons. A shared nothing design is likely to perform better with a simpler implementation

You probably shouldn't share epoll FDs across threads for performance reasons. A shared nothing design is likely to perform better with a simpler implementation in both the application and the kernel. I don't see sharing FDs across threads as a useful thing to aspire to. The common design I see these days is load balancing FDs across shared nothing threads. The thread that receives the notification via the selector is the thread that does the IO (no other thread has that FD). Keep adding threads as makes sense and never block let them block. A combined queue makes sense when task sizes are large. For small tasks the performance is poor. I see the queuing decision as something you make after you have already retrieved the message from the network and presented it to an application layer which makes a decision on how to dispatch. It would be cool if the kernel would do this for you under the hood! That would be amazing. Just making it correct is not enough though.

nine_k 10y ago
If no data are shared, what is the point of having threads, as opposed to processes?
- tyingq 10y ago
  One example is in NGINX. They mostly follow a "no threads" pattern, but specifically for Linux, they support "some threads" to get around issues with Linux aio. https://www.nginx.com/blog/thread-pools-boost-performance-9x/ https://www.nginx.com/blog/thread-pools-boost-performance-9x... That's not "shared nothing" threads, of course, so it doesn't answer your question at a high level. It does highlight, though, that Linux is especially challenging in this space as compared to FreeBSD. And not just because of epoll().
- arielweisberg 10y ago
  FD =\= data Get the data from the FD and then let the application decide the best dispatch policy.
- redahs 10y ago
  "Message passing" might be considered a component of a "shared nothing" system. If so, a thread-based message passing implementation might still use shared process memory behind the scenes to efficiently pass messages between local threads. Worker threads might be written so that the responsibility for managing a resource external to the process is assigned to max 1 thread simultaneously, even if the message passing abstraction uses a bit of shared process memory behind the scenes.
- BuuQu9hu 10y ago
  Threads are still the best way to get asynchronous filesystem I/O in a reliable way. I know that people will reply with various explanations of how to do it for some cases some of the time; however, nobody will post a way that works as reliably as threads, because none exist. At the same time, threads are a vile abomination and we should shun them whenever possible. Computers are terrible, aren't they?
redahs 10y ago
Yes, in a multithreaded scenario it's possible to think of poll() and epoll() primarily as an alternative to pthread_cond_timedwait(). These functions are not about 'event dispatching' so much as they are about putting the current thread to sleep until something of interest might have occurred. Using poll() or epoll() means that the thread gets to wake up on file descriptor events in addition to intrathread signals, and epoll() seems to work fine in this regard.

nine_k 10y ago

If no data are shared, what is the point of having threads, as opposed to processes?

tyingq 10y ago
One example is in NGINX. They mostly follow a "no threads" pattern, but specifically for Linux, they support "some threads" to get around issues with Linux aio. https://www.nginx.com/blog/thread-pools-boost-performance-9x/ https://www.nginx.com/blog/thread-pools-boost-performance-9x... That's not "shared nothing" threads, of course, so it doesn't answer your question at a high level. It does highlight, though, that Linux is especially challenging in this space as compared to FreeBSD. And not just because of epoll().
arielweisberg 10y ago
FD =\= data Get the data from the FD and then let the application decide the best dispatch policy.
redahs 10y ago
"Message passing" might be considered a component of a "shared nothing" system. If so, a thread-based message passing implementation might still use shared process memory behind the scenes to efficiently pass messages between local threads. Worker threads might be written so that the responsibility for managing a resource external to the process is assigned to max 1 thread simultaneously, even if the message passing abstraction uses a bit of shared process memory behind the scenes.
BuuQu9hu 10y ago
Threads are still the best way to get asynchronous filesystem I/O in a reliable way. I know that people will reply with various explanations of how to do it for some cases some of the time; however, nobody will post a way that works as reliably as threads, because none exist. At the same time, threads are a vile abomination and we should shun them whenever possible. Computers are terrible, aren't they?

redahs 10y ago

Yes, in a multithreaded scenario it's possible to think of poll() and epoll() primarily as an alternative to pthread_cond_timedwait(). These functions are not about 'event dispatching' so much as they are about putting the current thread to sleep until something of interest might have occurred. Using poll() or epoll() means that the thread gets to wake up on file descriptor events in addition to intrathread signals, and epoll() seems to work fine in this regard.