4 ms·

What where your performance requirements, what where some issues that you ran into, and what did you do to improve?

Const-me 4y ago
It was years ago, but as far as I remember here's the hard parts. Efficient synchronization is hard. Tasks may spawn more tasks, i.e. the pool has multiple producers and consumers. With a few cores simple ways work fine, with a few dozen of cores a global queue is a bottleneck. Dispatching too many tasks, fire & forget style, may consume too much memory with pending tasks. Thread pools often need a backpressure, i.e. a way to stop posting tasks until some are finished. Many articles on the internets promote lockless data structures, they have 2 issues. (1) very hard to implement and especially debug (2) the locks are still there in hardware, only their API is much worse (compare+exchange atomics), and the performance ain't necessarily better because reading from a cache line recently modified by another core is expensive, even more so than a cache miss and system RAM latency. Good automatic scaling is hard. All operating systems have APIs to find count of hardware threads which is good, but they all run multiple programs at once. Just because a computer has 8 hardware threads doesn't mean it's always a good idea for a particular program to spawn 8 threads to compute something, depending on the environment could be too many. Another notable mention is a tradeoff between CPU overhead and latency. Ideally you'd want neither of these things, but that's often not really possible. What's worse, optimal tradeoff varies: for large count of short tasks latency can be important, a background task which take a minute to complete doesn't care about latency at all, on a laptop it should ideally depend on plugged/unplugged global state. As to what helped, kernel APIs for sending messages across threads did, an important piece of the puzzle already implemented in the OSes. For Linux see mq_overview, poll, epoll; for Windows PostQueuedCompletionStatus and GetQueuedCompletionStatus.
- eklitzke 4y ago
  I personally found that when implementing a thread pool just using a regular mutex/condition variable and std::queue was just as performant as boost::lockfree::queue, was a lot simpler, and introduced fewer dependencies. Also when using boost::lockfree::queue I got complaints from tsan about some of the things boost was doing. I didn't really dig in but even if you assume that the boost implementation is correct, the fact that it pollutes tsan output is really annoying. There are probably cases where a lockfree queue makes sense, but I don't think a thread work pool is one of them. Using a mutex also gives you the option for doing batched inserts efficiently (acquire mutex, insert multiple items into the pool).

It was years ago, but as far as I remember here's the hard parts. Efficient synchronization is hard. Tasks may spawn more tasks, i.e. the pool has multiple producers and consumers. With a few cores simple ways work fine, with a few dozen of cores a global queue is a bottleneck. Dispatching too many tasks, fire & forget style, may consume too much memory with pending tasks. Thread pools often need a backpressure, i.e. a way to stop posting tasks until some are finished. Many articles on the internets promote lockless data structures, they have 2 issues. (1) very hard to implement and especially debug (2) the locks are still there in hardware, only their API is much worse (compare+exchange atomics), and the performance ain't necessarily better because reading from a cache line recently modified by another core is expensive, even more so than a cache miss and system RAM latency. Good automatic scaling is hard. All operating systems have APIs to find count of hardware threads which is good, but they all run multiple programs at once. Just because a computer has 8 hardware threads doesn't mean it's always a good idea for a particular program to spawn 8 threads to compute something, depending on the environment could be too many. Another notable mention is a tradeoff between CPU overhead and latency. Ideally you'd want neither of these things, but that's often not really possible. What's worse, optimal tradeoff varies: for large count of short tasks latency can be important, a background task which take a minute to complete doesn't care about latency at all, on a laptop it should ideally depend on plugged/unplugged global state. As to what helped, kernel APIs for sending messages across threads did, an important piece of the puzzle already implemented in the OSes. For Linux see mq_overview, poll, epoll; for Windows PostQueuedCompletionStatus and GetQueuedCompletionStatus.

eklitzke 4y ago
I personally found that when implementing a thread pool just using a regular mutex/condition variable and std::queue was just as performant as boost::lockfree::queue, was a lot simpler, and introduced fewer dependencies. Also when using boost::lockfree::queue I got complaints from tsan about some of the things boost was doing. I didn't really dig in but even if you assume that the boost implementation is correct, the fact that it pollutes tsan output is really annoying. There are probably cases where a lockfree queue makes sense, but I don't think a thread work pool is one of them. Using a mutex also gives you the option for doing batched inserts efficiently (acquire mutex, insert multiple items into the pool).