3 ms·

If I understand the paper correctly it's because GPUs are good at doing a lot of parallel matrix multiplications, at the cost of copying to and from GPU memory.

by carlmr 6y ago

If I understand the paper correctly it's because GPUs are good at doing a lot of parallel matrix multiplications, at the cost of copying to and from GPU memory. The CPU has the advantage that it doesn't need to copy, and it's more flexible, but not as parallel, so if you use caching and limit the parallelism of the computations by exploiting sparsity the CPU wins as it does here. So I don't see the clickbaitiness here, it's the central claim of the paper.

dragontamer 6y ago
If you are training on the GPU, the bulk of the weight data never leaves the GPU in the first place. It's a fundamentally different programming style, which makes comparisons extremely difficult. Furthermore: you hide latency on the GPU by running more and more parallel instances. Yeah, there is a 5 to 10 microsecond delay in all CPU to GPU comms and back, but just double, triple, or 100x fold the parallelism and batch up more tasks (aka Gustafson's law). Don't run one SLIDE. Run 100x of them in parallel to hide the latency. AMD Vega 64 has 16384 hardware threads (one instruction every 4 clock ticks across 4096 hardware SIMD cores) with up to 10 occupancy. That's 163840 max hardware SIMD threads of conceptual execution. (Occupancy is similar to hyperthreads/SMT). In practice, you run out of VRAM before running out of threads typically. 1000x instances uses 1000x the RAM. --------- Anyway, I've seen an entire generation of programmers try to make CPU only algorithms and only fail year after year. The cryptocoin community would generally prefer CPU mining rather than GPU or ASIC mining. But time and time again, the algorithms adapt and someone is brilliant enough to make the new alleged CPU algorithm work really well on a GPU.
- stjohnswarts 6y ago
  It's hard to compare in general but it's not hard to compare when you can take the same problem and run it in the "blackbox" of each system and compare performance. That's actually really easy. Now whether you can say "in general" would require much more varied comparisons than just a couple in an academic paper. Most people don't care what's in the black box.
  - dragontamer 6y ago
    I mean, I can invent an algorithm that runs poorly on x86 CPUs but good on GPUs rather easily. Ex: GPUs have a bit-reverse instruction, but x86 is missing that instruction. I've invented hash algorithms that only work well on GPUs (or really, architectures with single-cycle bit-reverse) and will run poorly on CPUs. ---------- But when I run the idea on a CPU, I don't do bitreverse. I use bswap64 instead. Because x86 has single-cycle bswap. Its technically a different algorithm with a different result, but bswap shuffles the bits around enough that the hash-algorithm is still really fast and really good (passes a lot of randomization tests I've done). ---------- We're looking at instruction-level differences between two grossly different platforms. Anyone who is an expert at both systems will tell you that optimization and tuning for both systems are so grossly different, that its pretty much non-sensical to try to compare them in "black-box" manners. You need to put some level of effort into processor-level tuning if you want the results to stand.

by carlmr 6y ago

dragontamer 6y ago

If you are training on the GPU, the bulk of the weight data never leaves the GPU in the first place. It's a fundamentally different programming style, which makes comparisons extremely difficult. Furthermore: you hide latency on the GPU by running more and more parallel instances. Yeah, there is a 5 to 10 microsecond delay in all CPU to GPU comms and back, but just double, triple, or 100x fold the parallelism and batch up more tasks (aka Gustafson's law). Don't run one SLIDE. Run 100x of them in parallel to hide the latency. AMD Vega 64 has 16384 hardware threads (one instruction every 4 clock ticks across 4096 hardware SIMD cores) with up to 10 occupancy. That's 163840 max hardware SIMD threads of conceptual execution. (Occupancy is similar to hyperthreads/SMT). In practice, you run out of VRAM before running out of threads typically. 1000x instances uses 1000x the RAM. --------- Anyway, I've seen an entire generation of programmers try to make CPU only algorithms and only fail year after year. The cryptocoin community would generally prefer CPU mining rather than GPU or ASIC mining. But time and time again, the algorithms adapt and someone is brilliant enough to make the new alleged CPU algorithm work really well on a GPU.

stjohnswarts 6y ago
It's hard to compare in general but it's not hard to compare when you can take the same problem and run it in the "blackbox" of each system and compare performance. That's actually really easy. Now whether you can say "in general" would require much more varied comparisons than just a couple in an academic paper. Most people don't care what's in the black box.
- dragontamer 6y ago
  I mean, I can invent an algorithm that runs poorly on x86 CPUs but good on GPUs rather easily. Ex: GPUs have a bit-reverse instruction, but x86 is missing that instruction. I've invented hash algorithms that only work well on GPUs (or really, architectures with single-cycle bit-reverse) and will run poorly on CPUs. ---------- But when I run the idea on a CPU, I don't do bitreverse. I use bswap64 instead. Because x86 has single-cycle bswap. Its technically a different algorithm with a different result, but bswap shuffles the bits around enough that the hash-algorithm is still really fast and really good (passes a lot of randomization tests I've done). ---------- We're looking at instruction-level differences between two grossly different platforms. Anyone who is an expert at both systems will tell you that optimization and tuning for both systems are so grossly different, that its pretty much non-sensical to try to compare them in "black-box" manners. You need to put some level of effort into processor-level tuning if you want the results to stand.