5 ms·

So wouldn't this be like 2.6 TFLOPS? I'm thinking if this can replace NVidia V100s to train something like ImageNet purely on CPU. However, V100 has 100 TFLOPS

So wouldn't this be like 2.6 TFLOPS? I'm thinking if this can replace NVidia V100s to train something like ImageNet purely on CPU. However, V100 has 100 TFLOPS which seems 50x more than 3990X. Perhaps, I'm reading the specs wrong? PS: Although FLOPS is not a good way to measure these stuff, it's a good indication of possible upper bound for deep learning related computation.

nneonneo 7y ago
Nah, you’re probably not reading the specs wrong. GPU-type devices severely outclass CPUs in raw compute power. This has been true for years, and it’s why deep learning depends on them. But CPUs and GPUs fulfill very different niches computationally; GPUs are incredibly parallel but aren’t well-suited for serial tasks or tasks that require unpredictable branching/looping, which is pretty much exactly what CPUs are good at.
- josh2600 7y ago
  Think of it like this: CPU’s do general computing. They’re super flexible but if you have a specific workload you might be able to use a different piece of silicon to get more performance. GPU’s do more parallelized computing but they don’t do as many different operations. They’re really good at doing a small not super complex fast but massively parallelized (like updating an array of pixels on a screen, for example). TPU’s are even more parallelized but the operations they do are even more specific and often simpler than the operations GPUs do.
  - hutzlibu 7y ago
    Aren't CPU's also slower, because they have a security overhead? (checking memory access, etc. which the gpu does not (so much))?
    - shaklee3 7y ago
      That's not really why they're slower. CPUs are significantly more complicated. Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon.
      - dragontamer 7y ago
        > Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon. Those things make CPUs faster at sequential execution. In effect: CPUs are latency optimized. GPUs are bandwidth optimized.
        josh2600 7y ago
        This is it exactly. The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task. The tradeoff is flexibility for application specific performance. CPUs can do hella stuff but they can’t do a specific thing faster than specialty hardware.
        dragontamer 7y ago
        GPUs are surprisingly flexible. GPUs are a full on turing machine. The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread. -------- Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively? Since CPUs are latency optimized, of course CPUs should speculate. GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run. As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster. --------- What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel. When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively. But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.
- shaklee3 7y ago
  It's reading the specs wrong for apples to apples. OP is using the tensor core numbers, which is half precision and only for matrix multiplies. Operations that don't fit that will use the standard fp32/16 performance of the chip, which is around 13TFLOPS. still higher than 2, but nowhere near 100.
jedbrown 7y ago
64 cores * 2.9 GHz * 8 single-precision lanes * 2 issue * 2 (FMA) = 5.9 TF. This compares with 14 TF for V100 (costs more and needs a host). The 100 TOPS for V100 refers to reduced precision (which may or may not be useful in a given ML training or prediction scenario). The V100 has much (~9x) higher memory bandwidth, but also higher latency.
- dragontamer 7y ago
  * Radeon VII GPU is 14.2 TFlops for $600 right now. * NVidia RTX 2070 Super is 9 TFlops for $500 True, Radeon VII and RTX 2070 are "consumer" GPUs... but Threadripper is similarly a "consumer" CPU and commands a lower price as a result. "Enterprise" products cost more. EPYC costs more than Threadripper, V100 costs more than RTX 2070 Super. If you're aiming at maximum performance at minimum price, you use consumer hardware.
  - AstralStorm 7y ago
    You pay the price in locked down fp64 functionality. (Typically quarter or less of fp32 rather than expected half.)
    - dragontamer 7y ago
      Most people don't need double-precision. Similarly, Threadripper loses on RDIMMs, LRDIMMs, and have 1/2 the memory channels and 1/2 the PCIe lanes. Most people don't need that either. Of course consumer products have fewer features than enterprise products. The chip manufacturers need to leave some features for "enterprise". The general idea is to extract more wealth from the people who can afford it, while providing consumers the features they care about at a lower cost. ----- Really, the feature enterprise GPUs need seems to be SR-IOV, or other PCIe-splitting technologies. This allows a singular GPU to be split over many VMs. Double-precision floats is niche to the scientific fields, also "enterprise" but I don't think most enterprise customers use double-precision floats.
choppaface 7y ago
DDR4 has a bandwidth of about 25 Gigabytes per second. The memory on a V100 does about 900 Gigabytes per second. Cerebras has 9.6 Petabytes per second of memory bandwidth. For stochastic gradient descent, which typically requires high-frequency read/writes, memory bandwidth is crucial. For ImageNet, you're trying to run well over 1TB of pixels through the processing device as quickly as possible while the processor uses a few gigabytes of scratch space.
- shaklee3 7y ago
  DDR has a bandwidth of about 25GBps per channel. You can hit around 100-200GBps on Epyc processors if you're utilizing ram efficiently. GPUs tend to enforce programming models that ensure more sequential accesses, but CPU can do it too.
  - choppaface 7y ago
    oh that's true thanks! I knew GDDR had higher bandwidth but the gap seemed a little high when I looked it up
- dragontamer 7y ago
  These stats are true, but the CPU's biggest advantage is L1, L2, and L3 cache. In particular, the 3990x, 64-core Threadripper will have 256MBs of aggregate L3 cache, and 512kBs L2 cache per core (32MBs of L2 cache). Highly-optimized kernels may fit large portions of data within L3 cache and rarely even touch DDR4! Note: Each L3 cache is only 16MBs between 4-cores. It will take some tricky programming to split a model into 16MB chunks, but if it can be done, Threadripper would be crazy fast. True, GPUs have really fat VRAM to work with, but CPUs have really fat L3 cache and L2 cache to work with. And the CPU caches are coherent too, simplifying atomic code. GPUs do have "shared memory" and L2 caches, but they're far smaller than CPU-caches.
- m0zg 7y ago
  >> Cerebras has 9.6 Petabytes More accurately, Cerebras has 9.6 "bullshitobytes" per second. If you can't verify this, it doesn't exist. You could claim insane "bandwidth" by considering your register file to be your "memory". But that doesn't make it so.

nneonneo 7y ago

Nah, you’re probably not reading the specs wrong. GPU-type devices severely outclass CPUs in raw compute power. This has been true for years, and it’s why deep learning depends on them. But CPUs and GPUs fulfill very different niches computationally; GPUs are incredibly parallel but aren’t well-suited for serial tasks or tasks that require unpredictable branching/looping, which is pretty much exactly what CPUs are good at.

josh2600 7y ago
Think of it like this: CPU’s do general computing. They’re super flexible but if you have a specific workload you might be able to use a different piece of silicon to get more performance. GPU’s do more parallelized computing but they don’t do as many different operations. They’re really good at doing a small not super complex fast but massively parallelized (like updating an array of pixels on a screen, for example). TPU’s are even more parallelized but the operations they do are even more specific and often simpler than the operations GPUs do.
- hutzlibu 7y ago
  Aren't CPU's also slower, because they have a security overhead? (checking memory access, etc. which the gpu does not (so much))?
  - shaklee3 7y ago
    That's not really why they're slower. CPUs are significantly more complicated. Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon.
    - dragontamer 7y ago
      > Things like branch prediction, transactions, sophisticated prefetching all take up a lot of silicon. Those things make CPUs faster at sequential execution. In effect: CPUs are latency optimized. GPUs are bandwidth optimized.
      - josh2600 7y ago
        This is it exactly. The more specific you get on the circuit the less flexible it is and the more bandwidth you get at a specific task. The tradeoff is flexibility for application specific performance. CPUs can do hella stuff but they can’t do a specific thing faster than specialty hardware.
        dragontamer 7y ago
        GPUs are surprisingly flexible. GPUs are a full on turing machine. The thing is, GPUs have horrible latency characteristics compared to CPUs. Whenever a GPU "has to wait" for RAM, it only switches to another thread. In contrast, CPUs will search your thread for out-of-order work, speculative work, and even prefetch memory ("guessing" what memory needs to be fetched) to help speed up the thread. -------- Consider speculative execution. Lets say there is a 50% chance that an if-statement is actually executed. Should your hardware execute the if-statement speculatively? Since CPUs are latency optimized, of course CPUs should speculate. GPUs however, are bandwidth optimized. Instead of speculating on the if-statement, the GPU will task switch and operate on another thread. GPUs have 8x to 10x SMT, many many threads waiting to be run. As such, GPUs would rather "make progress on another thread" rather than speculate to make a particular thread faster. --------- What problems can be represented in terms of a ton-of-threads ? Well, many simple image processing algorithms operate on 1920 x 1080 pixel entries, which immediately provides 2,073,600 pixels... or ~2-million items that often can be processed in parallel. When you have ~2-million items of work (aka: "CUDA Threads") waiting, the GPU is the superior architecture. Its better to make progress on "waiting threads" than to execute speculatively. But if you're a CPU with latency-optimized characteristics, programmers would rather have that if-statement speculated. The 50% chance of saving latency is worth more to a CPU programmer.
shaklee3 7y ago
It's reading the specs wrong for apples to apples. OP is using the tensor core numbers, which is half precision and only for matrix multiplies. Operations that don't fit that will use the standard fp32/16 performance of the chip, which is around 13TFLOPS. still higher than 2, but nowhere near 100.

jedbrown 7y ago

64 cores * 2.9 GHz * 8 single-precision lanes * 2 issue * 2 (FMA) = 5.9 TF. This compares with 14 TF for V100 (costs more and needs a host). The 100 TOPS for V100 refers to reduced precision (which may or may not be useful in a given ML training or prediction scenario). The V100 has much (~9x) higher memory bandwidth, but also higher latency.

dragontamer 7y ago
* Radeon VII GPU is 14.2 TFlops for $600 right now. * NVidia RTX 2070 Super is 9 TFlops for $500 True, Radeon VII and RTX 2070 are "consumer" GPUs... but Threadripper is similarly a "consumer" CPU and commands a lower price as a result. "Enterprise" products cost more. EPYC costs more than Threadripper, V100 costs more than RTX 2070 Super. If you're aiming at maximum performance at minimum price, you use consumer hardware.
- AstralStorm 7y ago
  You pay the price in locked down fp64 functionality. (Typically quarter or less of fp32 rather than expected half.)
  - dragontamer 7y ago
    Most people don't need double-precision. Similarly, Threadripper loses on RDIMMs, LRDIMMs, and have 1/2 the memory channels and 1/2 the PCIe lanes. Most people don't need that either. Of course consumer products have fewer features than enterprise products. The chip manufacturers need to leave some features for "enterprise". The general idea is to extract more wealth from the people who can afford it, while providing consumers the features they care about at a lower cost. ----- Really, the feature enterprise GPUs need seems to be SR-IOV, or other PCIe-splitting technologies. This allows a singular GPU to be split over many VMs. Double-precision floats is niche to the scientific fields, also "enterprise" but I don't think most enterprise customers use double-precision floats.

choppaface 7y ago

DDR4 has a bandwidth of about 25 Gigabytes per second. The memory on a V100 does about 900 Gigabytes per second. Cerebras has 9.6 Petabytes per second of memory bandwidth. For stochastic gradient descent, which typically requires high-frequency read/writes, memory bandwidth is crucial. For ImageNet, you're trying to run well over 1TB of pixels through the processing device as quickly as possible while the processor uses a few gigabytes of scratch space.

shaklee3 7y ago
DDR has a bandwidth of about 25GBps per channel. You can hit around 100-200GBps on Epyc processors if you're utilizing ram efficiently. GPUs tend to enforce programming models that ensure more sequential accesses, but CPU can do it too.
- choppaface 7y ago
  oh that's true thanks! I knew GDDR had higher bandwidth but the gap seemed a little high when I looked it up
dragontamer 7y ago
These stats are true, but the CPU's biggest advantage is L1, L2, and L3 cache. In particular, the 3990x, 64-core Threadripper will have 256MBs of aggregate L3 cache, and 512kBs L2 cache per core (32MBs of L2 cache). Highly-optimized kernels may fit large portions of data within L3 cache and rarely even touch DDR4! Note: Each L3 cache is only 16MBs between 4-cores. It will take some tricky programming to split a model into 16MB chunks, but if it can be done, Threadripper would be crazy fast. True, GPUs have really fat VRAM to work with, but CPUs have really fat L3 cache and L2 cache to work with. And the CPU caches are coherent too, simplifying atomic code. GPUs do have "shared memory" and L2 caches, but they're far smaller than CPU-caches.
m0zg 7y ago
>> Cerebras has 9.6 Petabytes More accurately, Cerebras has 9.6 "bullshitobytes" per second. If you can't verify this, it doesn't exist. You could claim insane "bandwidth" by considering your register file to be your "memory". But that doesn't make it so.