4 ms·

> Most threads are waiting for something – this is not the ideal activity for a thread. A thread should run (useful) code. Seems like that is an ideal activity

> Most threads are waiting for something – this is not the ideal activity for a thread. A thread should run (useful) code. Seems like that is an ideal activity for a thread, and in fact why most people use them? Combining all those waits into a single thread/process means also creating a dispatcher which then either spins up a thread to do work (ahem), or blocks all other useful work. If the cost is low (and for thread it usually is), having lots of blocked threads often makes the code easier to write and maintain.

macintux 5y ago
Agreed, I don’t understand this post at all. This is where Moore’s Law has brought us: if we want better performance, we need to take advantage of parallelism. Thus more threads.
- tcfhgj 5y ago
  you can't have more parallelism by simply increasing #threads. Parallelism is bound by CPU cores. Threads need to scale with CPU cores, however simply increasing #threads doesn't help. You need to split tasks of your program such that more tasks can be executed in parallel while not adding more overhead than you gain by parallelism. Then you can distribute those tasks on as many threads as possible (while still being bound by #cores).
  - astrange 5y ago
    > you can't have more parallelism by simply increasing #threads. Parallelism is bound by CPU cores. It's bound by the slowest shared resource in the system (Amdahl's law). That could be disk I/O, memory bandwidth, or power budget.
    - Icathian 5y ago
      That isn't what Amdahl's law says. Amdahl's law is an equation describing the theoretical paralellizability of a given task. It doesn't have anything to do with systems bottlenecking on one resource vs another.
Terry_Roll 5y ago
> Seems like that is an ideal activity for a thread Agree, because spawning a new thread and killing a thread carries its own overhead so better to have the thread sleep or be paused by the OS and cpu scheduling. Its all about getting the balance right, but I have noticed dont net apps run their gui on a single thread instead of a window(s) linked to a thread, like in a multi document interface (MDI) setup.
KerrAvon 5y ago
It is the ideal activity for a thread, but you actually don't want too many threads hanging around because each thread costs additional memory (stack + bookkeeping) and some amount of bookkeeping complexity for the OS scheduler, assuming native threads. That cost can really add up over the entire system.
- rndgermandude 5y ago
  How many is "too many" tho, realistically? In an OS like Windows, on "typical" hardware, the amount of memory per thread for stacks and bookkeeping the OS and/or userland allocate is usually negligible. In this context also remember that the reserved stack sizes per thread do not refer to physical memory but virtual memory (or "committed" vs "reserved"). A typical thread usually commits 1 page for the stack and the equivalent of up to 2 pages worth for any bookkeeping[0], etc, which would put the memory committed for the 7412 threads the author mentions in the ballpark of maybe 30-100 MiB of memory actually committed. The OS scheduler should be good enough to distinguish between runnable/waiting (sleeping) threads, eliminating the need to even look at most threads (which will be in waiting state) until some signal arrives putting them back into a runnable state. You only really get a problem when you have too many runnable threads (fighting over the available resources) or when too many threads keep rapidly toggling between runnable and waitable, as either case would potentially lead to a lot of expensive context switches (and the former case is an indication you're trying to do too much with the physical resources available). [0] Tho some things may commit considerably larger amounts of stack space and/or bookkeeping space. E.g. if I remember correctly java8 commits 1MiB of stack space per thread by default, while e.g. java11 merely reserves that 1MiB.
  - akvadrako 5y ago
    A beefy Linux server can handles a million threads fine. Of course, most of them need to be idle or you're going to be maxing out some other resource. The number of threads should only become a bottleneck if you have less than 0.01% of them idle.
  - bigbizisverywyz 5y ago
    The standard approach on Windows (and no reason also for linux, FreeBSD) is to have per-process thread pools. E.g. each process in a .net app has a default thread pool. Windows also has an async IO api that requires a thread pool of threads to handle the IO (handled in the Win32 API, so built into the core OS). A single thread can then wait for events that would require some processing (IO or timers being the most common) and then it passes that unit of work to the thread pool to be processed. Windows even takes pains to ensure that the last thread to process a unit of work, might then also be used to process the next item in the queue of work to eliminate costly context switching, and invalidation of data/instruction cache. Thread pools allow the best use of threading by: - limiting the number of threads a process creates/maintains. The thread pool itself can grow or shrink as demand changes. - limiting the number of threads that are in a runnable state by queueing work to the pool. The pool itself will be sized according to number of CPU cores, e.g. 2 * CPU cores. The main idea behind tread pools is that any thread that's not doing useful work is a waste of OS resources, so they make sure that they are either always fed with work, or get destroyed when idle for too long. From an overall machine efficiency perspective this makes total sense. Having hundred or even thousands of threads created and idle is massively wasteful and bad design, the OS should not be burdened with such housekeeping. Note that Apple's GCD also works on the same principle, as does libev/libuv.
jandrewrogers 5y ago
Switching between threads is expensive, so ideally the threads are never waiting unless the system is literally idle as that would imply a context switch. For software that has a lot of fine-grained multi-thread coordination, like database engines, a substantial percentage of all CPU time is spent coordinating and context-switching between threads, not actual workload. The more cores you have, the worse this issue becomes. This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance, especially on systems with lots of cores. Similarly, there is no requirement to spawn separate threads for task concurrency. The non-blocking coroutine-style concurrency common in high-performance server software supports much higher concurrency and with context switching costs that are much lower. The thread-per-core coroutine-style is easier to write and maintain too -- the logic is almost entirely single-threaded. No deadlocks, races, etc to worry about because lock graph concurrency is complicated.
- Diggsey 5y ago
  > ideally the threads are never waiting unless the system is literally idle as that would imply a context switch It wouldn't. There will only be a context switch if the thread is woken up for some reason (eg. the operation it's waiting on compeltes). If a thread is spending a majority of its time waiting, then it's not going to be doing a lot of context switches. > This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance That's a stretch. There are some specific real-time usecases where you want to lock a specific thread to each core, but those are extremely niche and imply total control over what is running on the system.
  - jandrewrogers 5y ago
    These architectures are used for throughput, which is much higher than classic multithreaded architectures, not real-time per se. Though they also can have very good latency characteristics if you sacrifice a bit of throughput. This architecture has been idiomatic for high-performance server software for a solid decade. It only seems niche if you don't work in an area of software that cares a lot about performance. The performance difference is not small, 10x throughput on the same hardware is not uncommon. And yes, it implies control of the resources your process nominally owns, though by no means the entire machine necessarily. Again, idiomatic, and software like high-end database engines have been designed this way for a very long time so it isn't a new idea. Designing software that takes full control of its resources, and does its own I/O and process scheduling, isn't magic, just different. A classic multithreaded software architecture will never be remotely performance competitive with a thread-per-core architecture. It isn't expected in theory, and is never demonstrated in practice. Modern thread-per-core architecture has its origins in supercomputing, specifically to enable better scalability as the number of cores per CPU (and server) became greater than 2.

macintux 5y ago

Agreed, I don’t understand this post at all. This is where Moore’s Law has brought us: if we want better performance, we need to take advantage of parallelism. Thus more threads.

tcfhgj 5y ago
you can't have more parallelism by simply increasing #threads. Parallelism is bound by CPU cores. Threads need to scale with CPU cores, however simply increasing #threads doesn't help. You need to split tasks of your program such that more tasks can be executed in parallel while not adding more overhead than you gain by parallelism. Then you can distribute those tasks on as many threads as possible (while still being bound by #cores).
- astrange 5y ago
  > you can't have more parallelism by simply increasing #threads. Parallelism is bound by CPU cores. It's bound by the slowest shared resource in the system (Amdahl's law). That could be disk I/O, memory bandwidth, or power budget.
  - Icathian 5y ago
    That isn't what Amdahl's law says. Amdahl's law is an equation describing the theoretical paralellizability of a given task. It doesn't have anything to do with systems bottlenecking on one resource vs another.

Terry_Roll 5y ago

> Seems like that is an ideal activity for a thread Agree, because spawning a new thread and killing a thread carries its own overhead so better to have the thread sleep or be paused by the OS and cpu scheduling. Its all about getting the balance right, but I have noticed dont net apps run their gui on a single thread instead of a window(s) linked to a thread, like in a multi document interface (MDI) setup.

KerrAvon 5y ago

It is the ideal activity for a thread, but you actually don't want too many threads hanging around because each thread costs additional memory (stack + bookkeeping) and some amount of bookkeeping complexity for the OS scheduler, assuming native threads. That cost can really add up over the entire system.

rndgermandude 5y ago
How many is "too many" tho, realistically? In an OS like Windows, on "typical" hardware, the amount of memory per thread for stacks and bookkeeping the OS and/or userland allocate is usually negligible. In this context also remember that the reserved stack sizes per thread do not refer to physical memory but virtual memory (or "committed" vs "reserved"). A typical thread usually commits 1 page for the stack and the equivalent of up to 2 pages worth for any bookkeeping[0], etc, which would put the memory committed for the 7412 threads the author mentions in the ballpark of maybe 30-100 MiB of memory actually committed. The OS scheduler should be good enough to distinguish between runnable/waiting (sleeping) threads, eliminating the need to even look at most threads (which will be in waiting state) until some signal arrives putting them back into a runnable state. You only really get a problem when you have too many runnable threads (fighting over the available resources) or when too many threads keep rapidly toggling between runnable and waitable, as either case would potentially lead to a lot of expensive context switches (and the former case is an indication you're trying to do too much with the physical resources available). [0] Tho some things may commit considerably larger amounts of stack space and/or bookkeeping space. E.g. if I remember correctly java8 commits 1MiB of stack space per thread by default, while e.g. java11 merely reserves that 1MiB.
- akvadrako 5y ago
  A beefy Linux server can handles a million threads fine. Of course, most of them need to be idle or you're going to be maxing out some other resource. The number of threads should only become a bottleneck if you have less than 0.01% of them idle.
- bigbizisverywyz 5y ago
  The standard approach on Windows (and no reason also for linux, FreeBSD) is to have per-process thread pools. E.g. each process in a .net app has a default thread pool. Windows also has an async IO api that requires a thread pool of threads to handle the IO (handled in the Win32 API, so built into the core OS). A single thread can then wait for events that would require some processing (IO or timers being the most common) and then it passes that unit of work to the thread pool to be processed. Windows even takes pains to ensure that the last thread to process a unit of work, might then also be used to process the next item in the queue of work to eliminate costly context switching, and invalidation of data/instruction cache. Thread pools allow the best use of threading by: - limiting the number of threads a process creates/maintains. The thread pool itself can grow or shrink as demand changes. - limiting the number of threads that are in a runnable state by queueing work to the pool. The pool itself will be sized according to number of CPU cores, e.g. 2 * CPU cores. The main idea behind tread pools is that any thread that's not doing useful work is a waste of OS resources, so they make sure that they are either always fed with work, or get destroyed when idle for too long. From an overall machine efficiency perspective this makes total sense. Having hundred or even thousands of threads created and idle is massively wasteful and bad design, the OS should not be burdened with such housekeeping. Note that Apple's GCD also works on the same principle, as does libev/libuv.

jandrewrogers 5y ago

Switching between threads is expensive, so ideally the threads are never waiting unless the system is literally idle as that would imply a context switch. For software that has a lot of fine-grained multi-thread coordination, like database engines, a substantial percentage of all CPU time is spent coordinating and context-switching between threads, not actual workload. The more cores you have, the worse this issue becomes. This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance, especially on systems with lots of cores. Similarly, there is no requirement to spawn separate threads for task concurrency. The non-blocking coroutine-style concurrency common in high-performance server software supports much higher concurrency and with context switching costs that are much lower. The thread-per-core coroutine-style is easier to write and maintain too -- the logic is almost entirely single-threaded. No deadlocks, races, etc to worry about because lock graph concurrency is complicated.

Diggsey 5y ago
> ideally the threads are never waiting unless the system is literally idle as that would imply a context switch It wouldn't. There will only be a context switch if the thread is woken up for some reason (eg. the operation it's waiting on compeltes). If a thread is spending a majority of its time waiting, then it's not going to be doing a lot of context switches. > This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance That's a stretch. There are some specific real-time usecases where you want to lock a specific thread to each core, but those are extremely niche and imply total control over what is running on the system.
- jandrewrogers 5y ago
  These architectures are used for throughput, which is much higher than classic multithreaded architectures, not real-time per se. Though they also can have very good latency characteristics if you sacrifice a bit of throughput. This architecture has been idiomatic for high-performance server software for a solid decade. It only seems niche if you don't work in an area of software that cares a lot about performance. The performance difference is not small, 10x throughput on the same hardware is not uncommon. And yes, it implies control of the resources your process nominally owns, though by no means the entire machine necessarily. Again, idiomatic, and software like high-end database engines have been designed this way for a very long time so it isn't a new idea. Designing software that takes full control of its resources, and does its own I/O and process scheduling, isn't magic, just different. A classic multithreaded software architecture will never be remotely performance competitive with a thread-per-core architecture. It isn't expected in theory, and is never demonstrated in practice. Modern thread-per-core architecture has its origins in supercomputing, specifically to enable better scalability as the number of cores per CPU (and server) became greater than 2.