4 ms·

Nah, you’re probably not reading the specs wrong. GPU-type devices severely outclass CPUs in raw compute power. This has been true for years, and it’s why deep

Nah, you’re probably not reading the specs wrong. GPU-type devices severely outclass CPUs in raw compute power. This has been true for years, and it’s why deep learning depends on them. But CPUs and GPUs fulfill very different niches computationally; GPUs are incredibly parallel but aren’t well-suited for serial tasks or tasks that require unpredictable branching/looping, which is pretty much exactly what CPUs are good at.

josh2600 7y ago
Think of it like this: CPU’s do general computing. They’re super flexible but if you have a specific workload you might be able to use a different piece of silicon to get more performance. GPU’s do more parallelized computing but they don’t do as many different operations. They’re really good at doing a small not super complex fast but massively parallelized (like updating an array of pixels on a screen, for example). TPU’s are even more parallelized but the operations they do are even more specific and often simpler than the operations GPUs do.
- hutzlibu 7y ago
  Aren't CPU's also slower, because they have a security overhead? (checking memory access, etc. which the gpu does not (so much))?
  - shaklee3 7y ago
    That's not really why they're slower. CPUs are significantly more complicated. Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon.
    - dragontamer 7y ago
      > Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon. Those things make CPUs faster at sequential execution. In effect: CPUs are latency optimized. GPUs are bandwidth optimized.
      - josh2600 7y ago
        This is it exactly. The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task. The tradeoff is flexibility for application specific performance. CPUs can do hella stuff but they can’t do a specific thing faster than specialty hardware.
        dragontamer 7y ago
        GPUs are surprisingly flexible. GPUs are a full on turing machine. The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread. -------- Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively? Since CPUs are latency optimized, of course CPUs should speculate. GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run. As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster. --------- What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel. When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively. But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.
        einpoklum 7y ago
        Remember that these millions of items must actually be able to do work independently, i.e. you need very few (or regular and localized) dependencies between the data processed by each thread, so that threads can just wait on things like memory accesses rather than waiting on _other threads_.
shaklee3 7y ago
It's reading the specs wrong for apples to apples. OP is using the tensor core numbers, which is half precision and only for matrix multiplies. Operations that don't fit that will use the standard fp32/16 performance of the chip, which is around 13TFLOPS. still higher than 2, but nowhere near 100.

josh2600 7y ago

Think of it like this: CPU’s do general computing. They’re super flexible but if you have a specific workload you might be able to use a different piece of silicon to get more performance. GPU’s do more parallelized computing but they don’t do as many different operations. They’re really good at doing a small not super complex fast but massively parallelized (like updating an array of pixels on a screen, for example). TPU’s are even more parallelized but the operations they do are even more specific and often simpler than the operations GPUs do.

hutzlibu 7y ago
Aren't CPU's also slower, because they have a security overhead? (checking memory access, etc. which the gpu does not (so much))?
- shaklee3 7y ago
  That's not really why they're slower. CPUs are significantly more complicated. Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon.
  - dragontamer 7y ago
    > Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon. Those things make CPUs faster at sequential execution. In effect: CPUs are latency optimized. GPUs are bandwidth optimized.
    - josh2600 7y ago
      This is it exactly. The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task. The tradeoff is flexibility for application specific performance. CPUs can do hella stuff but they can’t do a specific thing faster than specialty hardware.
      - dragontamer 7y ago
        GPUs are surprisingly flexible. GPUs are a full on turing machine. The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread. -------- Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively? Since CPUs are latency optimized, of course CPUs should speculate. GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run. As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster. --------- What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel. When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively. But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.
        einpoklum 7y ago
        Remember that these millions of items must actually be able to do work independently, i.e. you need very few (or regular and localized) dependencies between the data processed by each thread, so that threads can just wait on things like memory accesses rather than waiting on _other threads_.

shaklee3 7y ago

It's reading the specs wrong for apples to apples. OP is using the tensor core numbers, which is half precision and only for matrix multiplies. Operations that don't fit that will use the standard fp32/16 performance of the chip, which is around 13TFLOPS. still higher than 2, but nowhere near 100.