3 ms·

Just skimmed through the article, since I'm just here to testify that the most important revelation for me on writing APIs was that you can put and epoll_fd in

Just skimmed through the article, since I'm just here to testify that the most important revelation for me on writing APIs was that you can put and epoll_fd in an epoll_fd. This allows the API to have e.g. a single epoll_fd that contains all outbound connections, timers, signalfds and inotifys mentioned in the articled. Then the e.g. daemon using the APIs can have an epoll_fd per library it is using and just be sitting in the epoll_wait loop ready to fire library_x_process() call when events arrive.

kentonv 4y ago
Another use case for this: Say you have a set of "jobs" each composed of many "tasks" (each waiting for some event). The "jobs" are able to run concurrently on different threads, but the "tasks" must not run concurrently with other tasks in the same job because they might share data structures without synchronization. (This is a pretty common pattern in a lot of big servers.) Now you want to make sure you utilize multiple cores effectively. The naive approaches are: 1. Create a thread per job, each waiting on its own epoll specific to the job. This may be expensive if there are many jobs, and could allow too much concurrency. 2. Have a single epoll and a pool of threads waiting on it. Each thread must lock a mutex for the job that owns the task it's going to run. But a thread could receive an event for a task belonging to a job that's already running on another thread, in which case it has to synchronize with that other thread somehow, which is a pain. Be careful not to create a situation where all threads are blocked on the mutex for one job while other jobs are starved. Epoll nesting presents a clean solution: 3. Create an epoll per job, plus an outer epoll that waits on other epolls. A pool of threads waits on the outer epoll, which signals when a per-job epoll becomes ready. The thread receiving that event then takes ownership of the per-job epoll until the event queue is empty.
- remram 4y ago
  > The thread receiving that event then takes ownership of the per-job epoll until the event queue is empty. What do you mean by that? Take that per-job epoll out of the outer epoll?
  - kentonv 4y ago
    I'm handwaving a little because I haven't actually built something like this yet. But I imagine you'd add the per-job epoll to the global epoll with EPOLLONESHOT, so that once an event is reported, it is unregistered with the global epoll. Whatever thread received that event then owns the job. When that thread decides there's nothing more to do, it adds the job epoll back to the global epoll with EPOLLONESHOT again.

kentonv 4y ago

Another use case for this: Say you have a set of "jobs" each composed of many "tasks" (each waiting for some event). The "jobs" are able to run concurrently on different threads, but the "tasks" must not run concurrently with other tasks in the same job because they might share data structures without synchronization. (This is a pretty common pattern in a lot of big servers.) Now you want to make sure you utilize multiple cores effectively. The naive approaches are: 1. Create a thread per job, each waiting on its own epoll specific to the job. This may be expensive if there are many jobs, and could allow too much concurrency. 2. Have a single epoll and a pool of threads waiting on it. Each thread must lock a mutex for the job that owns the task it's going to run. But a thread could receive an event for a task belonging to a job that's already running on another thread, in which case it has to synchronize with that other thread somehow, which is a pain. Be careful not to create a situation where all threads are blocked on the mutex for one job while other jobs are starved. Epoll nesting presents a clean solution: 3. Create an epoll per job, plus an outer epoll that waits on other epolls. A pool of threads waits on the outer epoll, which signals when a per-job epoll becomes ready. The thread receiving that event then takes ownership of the per-job epoll until the event queue is empty.

remram 4y ago
> The thread receiving that event then takes ownership of the per-job epoll until the event queue is empty. What do you mean by that? Take that per-job epoll out of the outer epoll?
- kentonv 4y ago
  I'm handwaving a little because I haven't actually built something like this yet. But I imagine you'd add the per-job epoll to the global epoll with EPOLLONESHOT, so that once an event is reported, it is unregistered with the global epoll. Whatever thread received that event then owns the job. When that thread decides there's nothing more to do, it adds the job epoll back to the global epoll with EPOLLONESHOT again.