14 ms·

As an observation, performance optimized code is almost always effectively single-threaded these days, even when using all the cores on a CPU to very efficientl

As an observation, performance optimized code is almost always effectively single-threaded these days, even when using all the cores on a CPU to very efficiently process workloads. Given this, it is not clear to me that Rust actually buys much when it comes to parallel programming for the purposes of performance. Is there another reason to focus on parallelism aside from performance? This reminds me of when I use to write supercomputing codes. Lots of programming language nerds would wonder why we didn’t use functional models to simplify concurrency and parallelism. Our code was typically old school C++ (FORTRAN was already falling out of use). The truth was that 1) the software architecture was explicitly single-threaded — some of the first modern thread-per-core designs — to maximize performance, obviating any concerns about mutability and concurrency and 2) the primary performance bottlenecks tended to be memory bandwidth, of which functional programming paradigms tend to be relatively wasteful compared to something like C++. Consequently, C++ was actually simpler and higher performance for massively parallel computation, counterintuitively.

rational_indian 6y ago
> As an observation, performance optimized code is almost always effectively single-threaded these days, even when using all the cores on a CPU to very efficiently process workloads. Why? Edit: Thanks for all the replies. It seems this applies to data-parallel workloads only. I'd use a GPU for this. An RTX 3090 has around ~10000 CUDA cores (10000 simultaneous operations) v/s just ~10 for CPUs.
- mhh__ 6y ago
  I think the issue is that the memory is the bottleneck in many applications (i.e. 2 loads per cycle, despite many more Functionanl units) and those workloads tend to be very non-embarassinglt parallel.
- tehjoker 6y ago
  Sometimes your job has few or no inter-task dependencies and so there's no need to share between threads, but there's a heck of a lot of work that needs to be completed.
  - hnuser123456 6y ago
    Essentially any significant task can be made multi threaded for any number of cores, it's just a lot of coding work.
    - dllthomas 6y ago
      There's a practical question of whether this is true for realistically encountered problems, with a sufficient threshold on both the size and utility of a "significant task" and with realistic numbers of cores. Without a requirement of utility it's easy to come up with counterexamples from math, eg "does the Collatz starting from Graham's number reach one?" Once you've exhausted cores that can be used for the actual arithmetic, you are still gated by the decision-making at each step so cores cannot work too far "ahead" of each other. There may well be much smarter things we can do than brute force, but that's not "just coding work" at that point. Theoretically, it doesn't hold - at some point you have split apart everything that can be split, and you are left with some essential chains of data dependency that cannot be further parallelized.
- jandrewrogers 6y ago
  Data locality is everything for computational throughput. Having all data private to a single core is extraordinarily efficient compared to sharing data, and particularly mutable data, across cores. This creates a new problem: how do you balance load across cores? What if the workload is not evenly distributed across the data held by each core? Real workloads are like this! Fortunately, over the last decade, architectures and techniques for dynamic cross-core load shedding have become smooth and efficient while introducing negligible additional inter-core coordination. At this point, it is a mature way of designing extremely high throughput software.
  - anonymousDan 6y ago
    Can you point to any references/resources summarizing the latest cross-core dynamic load shedding techniques? Are they old techniques just now being applied in practice, or has something new been proposed?
    - electricshampo1 6y ago
      I don't know about latest; but try look at Scheduling Parallel Programs by Work Stealing with Private Deques https://hal.inria.fr/file/index/docid/863028/filename/full.pdf https://hal.inria.fr/file/index/docid/863028/filename/full.p...
  - cwalv 6y ago
    I'd like to learn more about the architectures and techniques developed over the last decade for this; can you recommend a few links or keywords to search?
- deleted 6y ago
  [deleted]
oivey 6y ago
> As an observation, performance optimized code is almost always effectively single-threaded these days, even when using all the cores on a CPU to very efficiently process workloads. Not my experience at all. One big problem is that most languages in 2021 have very, very poor support for thread-based parallelism. It’s crazy how many languages make it hard to do basic data parallel tasks. That steers people toward writing single threaded code and/or trying to rely on process-based parallelism which is basically strictly worse.
- Galanwe 6y ago
  > rely on process-based parallelism which is basically strictly worse. Why is that worse? I very seldomly use threads for concurrency, it creates monolithic binaries that are hard to maintain, configure, and understand. I much prefer a process based architecture with mmap'd shared memories for interprocess communications.
  - snovv_crash 6y ago
    Doesn't this just move the hard part, the maintenance, configuration and understanding, to a different (and equally complex) abstraction layer? Personally, I'd prefer to have a single binary that I start with some arguments, then need to also have a launch script, probably in a different language, which needs to coordinate all the starting, stopping, shared state etc. But, most of my work has been at the workstation level. Maybe it's different once you start needing clusters? The issue is that workstations have grown extremely powerful over the last few years with 50+ cores, 1000s of GPU 'cores', and hundreds of GB of memory. Clusters now bring the same headaches, in hardware, of multiprocess design.
    - Galanwe 6y ago
      > Doesn't this just move the hard part, the maintenance, configuration and understanding, to a different (and equally complex) abstraction layer? That's true indeed, but I find it more manageable that way. To me, managing independent processes instead of threads is especially powerful when the lifecycle of the concurrent work can vary. A typical example that happens quite often is when you have some kind of producer/consumer application. Say, you need to receive some data from a socket, and then process it. If implemented with threads, it becomes quite messy very fast. You end up with a big blob binary that spawns whatever the hell it wants and need to engineer some complex configuration file to tell it how many workers you want, etc. You also need some notification to this process to tell him to increase or decrease its amount of workers, etc. With independent process, it can be much more manageable. You can have a "collector" process that reads the socket and places messages in a shared memory, and if you want more/less workers, you just spawn worker processes to read from the shared memory.
dnautics 6y ago
It's not a bandwidth issue. Functional programming (especially, say, actor systems) is better for organizing mental models of concurrency when your concurrency is coupled with communication between the components. For hpc, you're typically optimizing for gustafson scaling (versus amdhal scaling) where you are running multiple copies of the same, computationally costly linearly organized code with no coupling between instances except statistical aggregation of results, so there is no particular benefit to functional-style concurrency. (And some FPLs,like Julia, are perfectly good at hpc anyways)
- jandrewrogers 6y ago
  FWIW, most supercomputing looks nothing like map-reduce; only the most trivial problems look like that. In a data model sense, a lot of supercomputing is join operation intensive, hence why they spend big bucks on high-bandwidth low-latency interconnects. STREAM benchmarks were more predictive of real-world supercomputing code performance than LAPACK in the majority of case 15+ years ago and it became more biased toward the former with time. The codes I worked on were complex graph analysis, spatiotemporal behavioral analysis, a bit of geospatial environmental modeling, and in prehistoric times thermal and mass transport modeling. These codes (pretty much anything involving reality) are intrinsically tightly coupled across compute nodes. Low-latency interconnects eventually gave way to latency-hiding software architectures but at no point did we use map-reduce as that would have been insanely inefficient given the sparsity and unpredictability of the interactions between nodes. These were the prototype software architectures for later high-performance databases. Every core is handling thousands or millions of independent shards of the larger computational model, which makes latency-hiding particularly efficient.
  - physicsguy 6y ago
    This is similar to my experience too. If people can write out a single Python function and apply it to all of a large amount of data then great, but that isn’t the majority of supercomputing programming.
    - dnautics 6y ago
trishume 6y ago
My impression is that what kind of parallelism patterns you need is pretty consistent within entire fields of programming. So you can go an entire career of performance optimization within HPC, game dev, film rendering or trading systems and never use the patterns the others say they use all the time. My experience with process-based parallelism is that yes on Linux it's basically isomorphic to thread-based parallelism. It's just so much more code to do the same thing. In Rust adding a new special-purpose background thread with some standard-library channels is 30 lines of code and I can probably even access the same logging system from the other thread. If I wanted to do that with processes I need to: - Coordinate a shared memory file over command line arguments or make sure everything is fork-safe - Find a library for shared-memory queues - Deal with making sure that if either process crashes the other process goes down with it in a reasonable way. - Make sure all my monitoring/logging is also hooked up to the other process. If I want to use a shared memory data-structure with atomics I need to either not use pointers or live dangerously and try and memory-map it at the exact same offsets in each process and ensure I use a special allocator for things in the shared file. Yes you can do all the same things with both approaches, I just find threads take way less code. It's not too bad if all your processes are doing the same thing, and you also need to scale to many servers anyhow. It's more annoying if you want to have a bunch of different types of special background processes.
- amboar 6y ago
  I realise your post is an argument in favour of Rust over C for these things, but regardless, you might be interested in a WIP library I've started to solve most of the issues you outlined: https://github.com/amboar/shmapper#libshmap https://github.com/amboar/shmapper#libshmap
- jez 6y ago
  > In Rust adding a new special-purpose background thread with some standard-library channels is 30 lines of code and I can probably even access the same logging system from the other thread. Do you happen to have a link to code that does this? This sounds similar to a problem I have right now and I’d love to see what solution you’ve arrived at.
- malkia 6y ago
_ph_ 6y ago
When we talk about support for threading/concurrent programming in programming languages, it is less about how to reach the theoretical limits of your system best, especially if you are free to architect the whole software stack towards that goal. In that case, your statements might apply. It is about how easily a programmer, who deals with a certain subtask in a system, can utilize more cores for the this task. Not talking about supercomputing, but looking at a smarktphone or a typical PC. There you usually have most cores just idle unused, but if the user triggers an action, you want to be able to use as many cores as it speeds up computation. Language support for parallelism makes a huge difference there. In Go I can write a function to do a certain computation and quite often it is trivial to spread several calls across goroutines.
- jorangreef 6y ago
  You are not factoring in the cost of context switches, and that many user applications today are memory-bound and not CPU-bound. It's one of the secrets exploited by the M1 chip, seen in how many more cache lines the CPU's LFB can fill concurrently compared to Intel chips and that these are now 128 byte cache lines instead of 64 byte cache lines.
  - _ph_ 6y ago
    Which context switches? With the Go model, I have exactly one thread per CPU, no context switches. And if you are memory-bound, why have more CPUs? But sure, there is a reason why the M1 has so stellar performance, it has one of the fastest single-thread performances and many applications do not manage to load more than 4 cores for common tasks - which partially is also a consequence of doing that is difficult in many programming languages, but easy in some, which are only slowly gaining traction.
    - jorangreef 6y ago
      > Which context switches? With the Go model, I have exactly one thread per CPU, no context switches. Not in the user application model you were describing. Those threads would need to coordinate and communicate (for example, back to the user interface), and that implies context switches. Unless you're thinking of a strictly isolated thread-per-core design (https://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html https://mechanical-sympathy.blogspot.com/2011/09/single-writ...), which means we are then in agreement. > And if you are memory-bound, why have more CPUs? Yes, exactly, and that's why parallelism can often make things worse: https://brooker.co.za/blog/2014/12/06/random.html https://brooker.co.za/blog/2014/12/06/random.html However, for independent processes, each additional CPU adds memory bandwidth (according to the NUMA model) because there's a concurrency limit to each CPU's LFB that puts an upper bound of 6 GB/s on filling cache lines for cache misses (even if the bandwidth of your memory system is actually much higher): https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_distributed/2018/slides/pdf/memory2.pdf https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...
bumbada 6y ago
It really depends of your definition of terms. What do you call "performance optimized"? For example I consider glyph drawing as "performance optimized". It requires massive parallelism just to be able to display text smoothly in a high definition screen. But most people will never see it, because they use a library that they call that does all the work for them and do not need to care about that. The difference is tremendous. We are talking 100x more efficiency just using GPUs alone. You can get 1000x, 10.000x with hardware(electronic chip design) acceleration parallelism(increasing the cost and rigidity, and times to market too). It is so big that it is a different level. It is not performance alone. It is that some things are so inefficient that are just not practical(like expending a million dollars in your energy bill in order to solve a problem). Same happens with of course 3D, audio or video recognition. Sensor I/O. Artificial intelligence. Rust lets you just prototype lots of code in a parallel way in the CPU, even for things that will run in a FPGA or ASIC in the future. It let's you transition smaller steps: CPU->GPU->FPGA->ASIC
gameswithgo 6y ago
>As an observation, performance optimized code is almost always effectively single-threaded these days I do not accept this premise. Things are increasingly multithreaded.

rational_indian 6y ago

> As an observation, performance optimized code is almost always effectively single-threaded these days, even when using all the cores on a CPU to very efficiently process workloads. Why? Edit: Thanks for all the replies. It seems this applies to data-parallel workloads only. I'd use a GPU for this. An RTX 3090 has around ~10000 CUDA cores (10000 simultaneous operations) v/s just ~10 for CPUs.

mhh__ 6y ago
I think the issue is that the memory is the bottleneck in many applications (i.e. 2 loads per cycle, despite many more Functionanl units) and those workloads tend to be very non-embarassinglt parallel.
tehjoker 6y ago
Sometimes your job has few or no inter-task dependencies and so there's no need to share between threads, but there's a heck of a lot of work that needs to be completed.
- hnuser123456 6y ago
  Essentially any significant task can be made multi threaded for any number of cores, it's just a lot of coding work.
  - dllthomas 6y ago
    There's a practical question of whether this is true for realistically encountered problems, with a sufficient threshold on both the size and utility of a "significant task" and with realistic numbers of cores. Without a requirement of utility it's easy to come up with counterexamples from math, eg "does the Collatz starting from Graham's number reach one?" Once you've exhausted cores that can be used for the actual arithmetic, you are still gated by the decision-making at each step so cores cannot work too far "ahead" of each other. There may well be much smarter things we can do than brute force, but that's not "just coding work" at that point. Theoretically, it doesn't hold - at some point you have split apart everything that can be split, and you are left with some essential chains of data dependency that cannot be further parallelized.
jandrewrogers 6y ago
Data locality is everything for computational throughput. Having all data private to a single core is extraordinarily efficient compared to sharing data, and particularly mutable data, across cores. This creates a new problem: how do you balance load across cores? What if the workload is not evenly distributed across the data held by each core? Real workloads are like this! Fortunately, over the last decade, architectures and techniques for dynamic cross-core load shedding have become smooth and efficient while introducing negligible additional inter-core coordination. At this point, it is a mature way of designing extremely high throughput software.
- anonymousDan 6y ago
  Can you point to any references/resources summarizing the latest cross-core dynamic load shedding techniques? Are they old techniques just now being applied in practice, or has something new been proposed?
  - electricshampo1 6y ago
    I don't know about latest; but try look at Scheduling Parallel Programs by Work Stealing with Private Deques https://hal.inria.fr/file/index/docid/863028/filename/full.pdf https://hal.inria.fr/file/index/docid/863028/filename/full.p...
- cwalv 6y ago
  I'd like to learn more about the architectures and techniques developed over the last decade for this; can you recommend a few links or keywords to search?
deleted 6y ago
[deleted]

oivey 6y ago

> As an observation, performance optimized code is almost always effectively single-threaded these days, even when using all the cores on a CPU to very efficiently process workloads. Not my experience at all. One big problem is that most languages in 2021 have very, very poor support for thread-based parallelism. It’s crazy how many languages make it hard to do basic data parallel tasks. That steers people toward writing single threaded code and/or trying to rely on process-based parallelism which is basically strictly worse.

Galanwe 6y ago
> rely on process-based parallelism which is basically strictly worse. Why is that worse? I very seldomly use threads for concurrency, it creates monolithic binaries that are hard to maintain, configure, and understand. I much prefer a process based architecture with mmap'd shared memories for interprocess communications.
- snovv_crash 6y ago
  Doesn't this just move the hard part, the maintenance, configuration and understanding, to a different (and equally complex) abstraction layer? Personally, I'd prefer to have a single binary that I start with some arguments, then need to also have a launch script, probably in a different language, which needs to coordinate all the starting, stopping, shared state etc. But, most of my work has been at the workstation level. Maybe it's different once you start needing clusters? The issue is that workstations have grown extremely powerful over the last few years with 50+ cores, 1000s of GPU 'cores', and hundreds of GB of memory. Clusters now bring the same headaches, in hardware, of multiprocess design.
  - Galanwe 6y ago
    > Doesn't this just move the hard part, the maintenance, configuration and understanding, to a different (and equally complex) abstraction layer? That's true indeed, but I find it more manageable that way. To me, managing independent processes instead of threads is especially powerful when the lifecycle of the concurrent work can vary. A typical example that happens quite often is when you have some kind of producer/consumer application. Say, you need to receive some data from a socket, and then process it. If implemented with threads, it becomes quite messy very fast. You end up with a big blob binary that spawns whatever the hell it wants and need to engineer some complex configuration file to tell it how many workers you want, etc. You also need some notification to this process to tell him to increase or decrease its amount of workers, etc. With independent process, it can be much more manageable. You can have a "collector" process that reads the socket and places messages in a shared memory, and if you want more/less workers, you just spawn worker processes to read from the shared memory.

dnautics 6y ago

It's not a bandwidth issue. Functional programming (especially, say, actor systems) is better for organizing mental models of concurrency when your concurrency is coupled with communication between the components. For hpc, you're typically optimizing for gustafson scaling (versus amdhal scaling) where you are running multiple copies of the same, computationally costly linearly organized code with no coupling between instances except statistical aggregation of results, so there is no particular benefit to functional-style concurrency. (And some FPLs,like Julia, are perfectly good at hpc anyways)

jandrewrogers 6y ago
FWIW, most supercomputing looks nothing like map-reduce; only the most trivial problems look like that. In a data model sense, a lot of supercomputing is join operation intensive, hence why they spend big bucks on high-bandwidth low-latency interconnects. STREAM benchmarks were more predictive of real-world supercomputing code performance than LAPACK in the majority of case 15+ years ago and it became more biased toward the former with time. The codes I worked on were complex graph analysis, spatiotemporal behavioral analysis, a bit of geospatial environmental modeling, and in prehistoric times thermal and mass transport modeling. These codes (pretty much anything involving reality) are intrinsically tightly coupled across compute nodes. Low-latency interconnects eventually gave way to latency-hiding software architectures but at no point did we use map-reduce as that would have been insanely inefficient given the sparsity and unpredictability of the interactions between nodes. These were the prototype software architectures for later high-performance databases. Every core is handling thousands or millions of independent shards of the larger computational model, which makes latency-hiding particularly efficient.
- physicsguy 6y ago
  This is similar to my experience too. If people can write out a single Python function and apply it to all of a large amount of data then great, but that isn’t the majority of supercomputing programming.
  - dnautics 6y ago

trishume 6y ago

My impression is that what kind of parallelism patterns you need is pretty consistent within entire fields of programming. So you can go an entire career of performance optimization within HPC, game dev, film rendering or trading systems and never use the patterns the others say they use all the time. My experience with process-based parallelism is that yes on Linux it's basically isomorphic to thread-based parallelism. It's just so much more code to do the same thing. In Rust adding a new special-purpose background thread with some standard-library channels is 30 lines of code and I can probably even access the same logging system from the other thread. If I wanted to do that with processes I need to: - Coordinate a shared memory file over command line arguments or make sure everything is fork-safe - Find a library for shared-memory queues - Deal with making sure that if either process crashes the other process goes down with it in a reasonable way. - Make sure all my monitoring/logging is also hooked up to the other process. If I want to use a shared memory data-structure with atomics I need to either not use pointers or live dangerously and try and memory-map it at the exact same offsets in each process and ensure I use a special allocator for things in the shared file. Yes you can do all the same things with both approaches, I just find threads take way less code. It's not too bad if all your processes are doing the same thing, and you also need to scale to many servers anyhow. It's more annoying if you want to have a bunch of different types of special background processes.

amboar 6y ago
I realise your post is an argument in favour of Rust over C for these things, but regardless, you might be interested in a WIP library I've started to solve most of the issues you outlined: https://github.com/amboar/shmapper#libshmap https://github.com/amboar/shmapper#libshmap
jez 6y ago
> In Rust adding a new special-purpose background thread with some standard-library channels is 30 lines of code and I can probably even access the same logging system from the other thread. Do you happen to have a link to code that does this? This sounds similar to a problem I have right now and I’d love to see what solution you’ve arrived at.
malkia 6y ago

_ph_ 6y ago

When we talk about support for threading/concurrent programming in programming languages, it is less about how to reach the theoretical limits of your system best, especially if you are free to architect the whole software stack towards that goal. In that case, your statements might apply. It is about how easily a programmer, who deals with a certain subtask in a system, can utilize more cores for the this task. Not talking about supercomputing, but looking at a smarktphone or a typical PC. There you usually have most cores just idle unused, but if the user triggers an action, you want to be able to use as many cores as it speeds up computation. Language support for parallelism makes a huge difference there. In Go I can write a function to do a certain computation and quite often it is trivial to spread several calls across goroutines.

jorangreef 6y ago
You are not factoring in the cost of context switches, and that many user applications today are memory-bound and not CPU-bound. It's one of the secrets exploited by the M1 chip, seen in how many more cache lines the CPU's LFB can fill concurrently compared to Intel chips and that these are now 128 byte cache lines instead of 64 byte cache lines.
- _ph_ 6y ago
  Which context switches? With the Go model, I have exactly one thread per CPU, no context switches. And if you are memory-bound, why have more CPUs? But sure, there is a reason why the M1 has so stellar performance, it has one of the fastest single-thread performances and many applications do not manage to load more than 4 cores for common tasks - which partially is also a consequence of doing that is difficult in many programming languages, but easy in some, which are only slowly gaining traction.
  - jorangreef 6y ago
    > Which context switches? With the Go model, I have exactly one thread per CPU, no context switches. Not in the user application model you were describing. Those threads would need to coordinate and communicate (for example, back to the user interface), and that implies context switches. Unless you're thinking of a strictly isolated thread-per-core design (https://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html https://mechanical-sympathy.blogspot.com/2011/09/single-writ...), which means we are then in agreement. > And if you are memory-bound, why have more CPUs? Yes, exactly, and that's why parallelism can often make things worse: https://brooker.co.za/blog/2014/12/06/random.html https://brooker.co.za/blog/2014/12/06/random.html However, for independent processes, each additional CPU adds memory bandwidth (according to the NUMA model) because there's a concurrency limit to each CPU's LFB that puts an upper bound of 6 GB/s on filling cache lines for cache misses (even if the bandwidth of your memory system is actually much higher): https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_distributed/2018/slides/pdf/memory2.pdf https://www.eidos.ic.i.u-tokyo.ac.jp/~tau/lecture/parallel_d...

bumbada 6y ago

It really depends of your definition of terms. What do you call "performance optimized"? For example I consider glyph drawing as "performance optimized". It requires massive parallelism just to be able to display text smoothly in a high definition screen. But most people will never see it, because they use a library that they call that does all the work for them and do not need to care about that. The difference is tremendous. We are talking 100x more efficiency just using GPUs alone. You can get 1000x, 10.000x with hardware(electronic chip design) acceleration parallelism(increasing the cost and rigidity, and times to market too). It is so big that it is a different level. It is not performance alone. It is that some things are so inefficient that are just not practical(like expending a million dollars in your energy bill in order to solve a problem). Same happens with of course 3D, audio or video recognition. Sensor I/O. Artificial intelligence. Rust lets you just prototype lots of code in a parallel way in the CPU, even for things that will run in a FPGA or ASIC in the future. It let's you transition smaller steps: CPU->GPU->FPGA->ASIC

gameswithgo 6y ago

>As an observation, performance optimized code is almost always effectively single-threaded these days I do not accept this premise. Things are increasingly multithreaded.