5 ms·

Writing a thread pool which is performant and scalable is surprisingly hard, I did couple times. And now, whenever I can, I tend to avoid doing that. Instead, I

Writing a thread pool which is performant and scalable is surprisingly hard, I did couple times. And now, whenever I can, I tend to avoid doing that. Instead, I prefer using the stuff already implemented by someone else, in OS userlands or runtime libraries. • C and C++ runtime libraries have OpenMP. Level of support varies across compilers but still, all mainstream ones support at least OpenMP 2.0. • .NET runtime comes with its own thread pool. The standard library provides both low-level APIs similar to the Windows’ built-in one, and higher-level things like Parallel.For remotely comparable to OpenMP. • Windows userland has another one, see CreateThreadpoolWork and SubmitThreadpoolWork APIs introduced in Vista. The API is totally different though, a programmer needs to manually submit each task to the pool. • iOS and OSX userlands also have a thread pool called Grand Central Dispatch. It’s very comparable to the Windows version.

jltsiren 4y ago
The main issue with OpenMP is that Apple ships a crippled version of Clang that does not support it by default. If you are developing on Mac, OpenMP should be considered an external dependency, with all the unpleasantness that means in a C++ project. If you have students writing code and the target environment is Linux, as is common in the academia, replacing OpenMP with something portable can be a good idea. Also, a thread pool that supports the equivalent of: #pragma omp parallel for schedule(static) and #pragma omp parallel for schedule(dynamic, 1) if often good enough for scientific computing. There is surprisingly little need for all the fancy things software engineers have invented because their threads have to communicate every minute.
- physicsguy 4y ago
  Pretty much everyone in scientific computing who uses a Mac will use Homebrew or Macports to install a standard version of GCC or Clang in my experience.
jstimpfle 4y ago
What where your performance requirements, what where some issues that you ran into, and what did you do to improve?
- Const-me 4y ago
  It was years ago, but as far as I remember here's the hard parts. Efficient synchronization is hard. Tasks may spawn more tasks, i.e. the pool has multiple producers and consumers. With a few cores simple ways work fine, with a few dozen of cores a global queue is a bottleneck. Dispatching too many tasks, fire & forget style, may consume too much memory with pending tasks. Thread pools often need a backpressure, i.e. a way to stop posting tasks until some are finished. Many articles on the internets promote lockless data structures, they have 2 issues. (1) very hard to implement and especially debug (2) the locks are still there in hardware, only their API is much worse (compare+exchange atomics), and the performance ain't necessarily better because reading from a cache line recently modified by another core is expensive, even more so than a cache miss and system RAM latency. Good automatic scaling is hard. All operating systems have APIs to find count of hardware threads which is good, but they all run multiple programs at once. Just because a computer has 8 hardware threads doesn't mean it's always a good idea for a particular program to spawn 8 threads to compute something, depending on the environment could be too many. Another notable mention is a tradeoff between CPU overhead and latency. Ideally you'd want neither of these things, but that's often not really possible. What's worse, optimal tradeoff varies: for large count of short tasks latency can be important, a background task which take a minute to complete doesn't care about latency at all, on a laptop it should ideally depend on plugged/unplugged global state. As to what helped, kernel APIs for sending messages across threads did, an important piece of the puzzle already implemented in the OSes. For Linux see mq_overview, poll, epoll; for Windows PostQueuedCompletionStatus and GetQueuedCompletionStatus.
  - eklitzke 4y ago
    I personally found that when implementing a thread pool just using a regular mutex/condition variable and std::queue was just as performant as boost::lockfree::queue, was a lot simpler, and introduced fewer dependencies. Also when using boost::lockfree::queue I got complaints from tsan about some of the things boost was doing. I didn't really dig in but even if you assume that the boost implementation is correct, the fact that it pollutes tsan output is really annoying. There are probably cases where a lockfree queue makes sense, but I don't think a thread work pool is one of them. Using a mutex also gives you the option for doing batched inserts efficiently (acquire mutex, insert multiple items into the pool).

jltsiren 4y ago

The main issue with OpenMP is that Apple ships a crippled version of Clang that does not support it by default. If you are developing on Mac, OpenMP should be considered an external dependency, with all the unpleasantness that means in a C++ project. If you have students writing code and the target environment is Linux, as is common in the academia, replacing OpenMP with something portable can be a good idea. Also, a thread pool that supports the equivalent of: #pragma omp parallel for schedule(static) and #pragma omp parallel for schedule(dynamic, 1) if often good enough for scientific computing. There is surprisingly little need for all the fancy things software engineers have invented because their threads have to communicate every minute.

physicsguy 4y ago
Pretty much everyone in scientific computing who uses a Mac will use Homebrew or Macports to install a standard version of GCC or Clang in my experience.

jstimpfle 4y ago

What where your performance requirements, what where some issues that you ran into, and what did you do to improve?

Const-me 4y ago
It was years ago, but as far as I remember here's the hard parts. Efficient synchronization is hard. Tasks may spawn more tasks, i.e. the pool has multiple producers and consumers. With a few cores simple ways work fine, with a few dozen of cores a global queue is a bottleneck. Dispatching too many tasks, fire & forget style, may consume too much memory with pending tasks. Thread pools often need a backpressure, i.e. a way to stop posting tasks until some are finished. Many articles on the internets promote lockless data structures, they have 2 issues. (1) very hard to implement and especially debug (2) the locks are still there in hardware, only their API is much worse (compare+exchange atomics), and the performance ain't necessarily better because reading from a cache line recently modified by another core is expensive, even more so than a cache miss and system RAM latency. Good automatic scaling is hard. All operating systems have APIs to find count of hardware threads which is good, but they all run multiple programs at once. Just because a computer has 8 hardware threads doesn't mean it's always a good idea for a particular program to spawn 8 threads to compute something, depending on the environment could be too many. Another notable mention is a tradeoff between CPU overhead and latency. Ideally you'd want neither of these things, but that's often not really possible. What's worse, optimal tradeoff varies: for large count of short tasks latency can be important, a background task which take a minute to complete doesn't care about latency at all, on a laptop it should ideally depend on plugged/unplugged global state. As to what helped, kernel APIs for sending messages across threads did, an important piece of the puzzle already implemented in the OSes. For Linux see mq_overview, poll, epoll; for Windows PostQueuedCompletionStatus and GetQueuedCompletionStatus.
- eklitzke 4y ago
  I personally found that when implementing a thread pool just using a regular mutex/condition variable and std::queue was just as performant as boost::lockfree::queue, was a lot simpler, and introduced fewer dependencies. Also when using boost::lockfree::queue I got complaints from tsan about some of the things boost was doing. I didn't really dig in but even if you assume that the boost implementation is correct, the fact that it pollutes tsan output is really annoying. There are probably cases where a lockfree queue makes sense, but I don't think a thread work pool is one of them. Using a mutex also gives you the option for doing batched inserts efficiently (acquire mutex, insert multiple items into the pool).