3 ms·

This is it exactly. The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task. The tradeoff is flexib

This is it exactly. The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task. The tradeoff is flexibility for application specific performance. CPUs can do hella stuff but they can’t do a specific thing faster than specialty hardware.

dragontamer 7y ago
GPUs are surprisingly flexible. GPUs are a full on turing machine. The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread. -------- Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively? Since CPUs are latency optimized, of course CPUs should speculate. GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run. As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster. --------- What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel. When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively. But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.
- einpoklum 7y ago
  Remember that these millions of items must actually be able to do work independently, i.e. you need very few (or regular and localized) dependencies between the data processed by each thread, so that threads can just wait on things like memory accesses rather than waiting on _other threads_.
  - dragontamer 7y ago
    > rather than waiting on _other threads_. Hmm, I think I see what you're trying to say, but maybe more precise language would be better here. GPU cores have extremely efficient thread-barrier instructions. NVidia PTX has "barrier", while AMD has "S_BARRIER". Both of which allow the ~256 threads of a workgroup to efficiently wait for each other. ------- The other aspect, is that "Waiting on Memory" (at least, waiting on L2 memory) is globally synchronized in both AMD and NVidia systems. Waiting for an L2 atomic operation to complete IS a synchronization event, because L2 cache has a total memory ordering on both AMD and NVidia platforms. Tying L2 cache to higher levels allows for memory coherence with the host CPU, or other GPUs even. That is to say: "dependencies" are often turned into memory-sync / memory-barrier events at the lowest level. Synchronizing threads is one-and-the-same as waiting on memory. (Specifically: creating a load-and-store ordering that all cores can agree upon). --------- I think what you're trying to say is that dependency chains must be short on GPUs, and that there are many parallel-threads of dependency chains to execute. In these circumstances, an algorithm can run efficiently. If you have an algorithm that is an explicit dependency chain from beginning to end (ironically: Ethereum hashing satisfies this constraint. You work on the same hash for millions of iterations...), then that particular dependency chain cannot be cut or parallelized. But Ethereum Hashing is still parallel, because there are trillions of guesses that can all work in parallel. So while its impossible to parallelize a singular ETH Hash... you can run many ETH Hashes in parallel with each other.

dragontamer 7y ago

GPUs are surprisingly flexible. GPUs are a full on turing machine. The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread. -------- Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively? Since CPUs are latency optimized, of course CPUs should speculate. GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run. As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster. --------- What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel. When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively. But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.

einpoklum 7y ago
Remember that these millions of items must actually be able to do work independently, i.e. you need very few (or regular and localized) dependencies between the data processed by each thread, so that threads can just wait on things like memory accesses rather than waiting on _other threads_.
- dragontamer 7y ago
  > rather than waiting on _other threads_. Hmm, I think I see what you're trying to say, but maybe more precise language would be better here. GPU cores have extremely efficient thread-barrier instructions. NVidia PTX has "barrier", while AMD has "S_BARRIER". Both of which allow the ~256 threads of a workgroup to efficiently wait for each other. ------- The other aspect, is that "Waiting on Memory" (at least, waiting on L2 memory) is globally synchronized in both AMD and NVidia systems. Waiting for an L2 atomic operation to complete IS a synchronization event, because L2 cache has a total memory ordering on both AMD and NVidia platforms. Tying L2 cache to higher levels allows for memory coherence with the host CPU, or other GPUs even. That is to say: "dependencies" are often turned into memory-sync / memory-barrier events at the lowest level. Synchronizing threads is one-and-the-same as waiting on memory. (Specifically: creating a load-and-store ordering that all cores can agree upon). --------- I think what you're trying to say is that dependency chains must be short on GPUs, and that there are many parallel-threads of dependency chains to execute. In these circumstances, an algorithm can run efficiently. If you have an algorithm that is an explicit dependency chain from beginning to end (ironically: Ethereum hashing satisfies this constraint. You work on the same hash for millions of iterations...), then that particular dependency chain cannot be cut or parallelized. But Ethereum Hashing is still parallel, because there are trillions of guesses that can all work in parallel. So while its impossible to parallelize a singular ETH Hash... you can run many ETH Hashes in parallel with each other.