3 ms·

Switching between threads is expensive, so ideally the threads are never waiting unless the system is literally idle as that would imply a context switch. For s

Switching between threads is expensive, so ideally the threads are never waiting unless the system is literally idle as that would imply a context switch. For software that has a lot of fine-grained multi-thread coordination, like database engines, a substantial percentage of all CPU time is spent coordinating and context-switching between threads, not actual workload. The more cores you have, the worse this issue becomes. This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance, especially on systems with lots of cores. Similarly, there is no requirement to spawn separate threads for task concurrency. The non-blocking coroutine-style concurrency common in high-performance server software supports much higher concurrency and with context switching costs that are much lower. The thread-per-core coroutine-style is easier to write and maintain too -- the logic is almost entirely single-threaded. No deadlocks, races, etc to worry about because lock graph concurrency is complicated.

Diggsey 5y ago
> ideally the threads are never waiting unless the system is literally idle as that would imply a context switch It wouldn't. There will only be a context switch if the thread is woken up for some reason (eg. the operation it's waiting on compeltes). If a thread is spending a majority of its time waiting, then it's not going to be doing a lot of context switches. > This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance That's a stretch. There are some specific real-time usecases where you want to lock a specific thread to each core, but those are extremely niche and imply total control over what is running on the system.
- jandrewrogers 5y ago
  These architectures are used for throughput, which is much higher than classic multithreaded architectures, not real-time per se. Though they also can have very good latency characteristics if you sacrifice a bit of throughput. This architecture has been idiomatic for high-performance server software for a solid decade. It only seems niche if you don't work in an area of software that cares a lot about performance. The performance difference is not small, 10x throughput on the same hardware is not uncommon. And yes, it implies control of the resources your process nominally owns, though by no means the entire machine necessarily. Again, idiomatic, and software like high-end database engines have been designed this way for a very long time so it isn't a new idea. Designing software that takes full control of its resources, and does its own I/O and process scheduling, isn't magic, just different. A classic multithreaded software architecture will never be remotely performance competitive with a thread-per-core architecture. It isn't expected in theory, and is never demonstrated in practice. Modern thread-per-core architecture has its origins in supercomputing, specifically to enable better scalability as the number of cores per CPU (and server) became greater than 2.
djsweet 5y ago
> No deadlocks, races, etc to worry about because lock graph concurrency is complicated. I’ve spent the last two and a half years mainly programming in single-threaded NodeJS environments with a focus on heavy same-task concurrency, and I’ve still run into deadlocks and races, despite the lack of OS threads involved. However, your point that context switching is expensive, and is why thread-per-core architectures are so common, still holds.

Diggsey 5y ago

> ideally the threads are never waiting unless the system is literally idle as that would imply a context switch It wouldn't. There will only be a context switch if the thread is woken up for some reason (eg. the operation it's waiting on compeltes). If a thread is spending a majority of its time waiting, then it's not going to be doing a lot of context switches. > This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance That's a stretch. There are some specific real-time usecases where you want to lock a specific thread to each core, but those are extremely niche and imply total control over what is running on the system.

jandrewrogers 5y ago
These architectures are used for throughput, which is much higher than classic multithreaded architectures, not real-time per se. Though they also can have very good latency characteristics if you sacrifice a bit of throughput. This architecture has been idiomatic for high-performance server software for a solid decade. It only seems niche if you don't work in an area of software that cares a lot about performance. The performance difference is not small, 10x throughput on the same hardware is not uncommon. And yes, it implies control of the resources your process nominally owns, though by no means the entire machine necessarily. Again, idiomatic, and software like high-end database engines have been designed this way for a very long time so it isn't a new idea. Designing software that takes full control of its resources, and does its own I/O and process scheduling, isn't magic, just different. A classic multithreaded software architecture will never be remotely performance competitive with a thread-per-core architecture. It isn't expected in theory, and is never demonstrated in practice. Modern thread-per-core architecture has its origins in supercomputing, specifically to enable better scalability as the number of cores per CPU (and server) became greater than 2.

djsweet 5y ago

> No deadlocks, races, etc to worry about because lock graph concurrency is complicated. I’ve spent the last two and a half years mainly programming in single-threaded NodeJS environments with a focus on heavy same-task concurrency, and I’ve still run into deadlocks and races, despite the lack of OS threads involved. However, your point that context switching is expensive, and is why thread-per-core architectures are so common, still holds.