4 ms·

I wonder how does it compare to running same model on the same cpu in ram(making sure fast cpu ML library that utilises AVX2 is used for example Intel MKL)? Al

I wonder how does it compare to running same model on the same cpu in ram(making sure fast cpu ML library that utilises AVX2 is used for example Intel MKL)? Also, when doing a test like this it's important to compare same bit depths so fp32 on both.

dragontamer 3y ago
My untested rule of thumb is that if something fits inside of GPU-register space, its probably faster on GPU. (GPUs have more architectural registers than CPUs, since CPUs are using all those "shadow registers" to implement out-of-order stuffs). Compilers will automatically turn your most-used variables into registers today, though you might need to verify by looking at the assembly code whether or not you're actually in register-space. But if something fits inside of CPU-cache space, its probably faster on CPU. (Intel is upto 2MB L2 cache on some systems, AMD is up to 192MB L3 cache on some systems). But if something is outside of CPU-cache space but fits inside of GPU-VRAM space, its probably faster on GPU. (Better to be bandwidth-limited at 500GB/s GPU-VRAM speeds than 50GB/s DDR5 speed). Ex: 16GBs GPU-VRAM is back into GPU-sized solutions to a problem. Then if something fits inside of CPU-RAM space, which is like 2TB in practice (!!!!), then CPUs are probably faster. Because there's no point taking a 2TB DDR5 RAM limited problem, passing some of that data to 16GB or 96GB of GPU-VRAM, and then waiting for DDR5 RAM anyway. ------------ Not that I've tested out this theory. But it'd make sense in my brain at least, lol.
- deleted 3y ago
  [deleted]
Const-me 3y ago
OP has tested the 4600G APU. Theoretically, the chip can do 710 FP32 GFlops on CPU, and 1702 GFlops on the integrated GPU. Another thing, it’s hard to write CPU code which saturates memory throughput instead of stalling on memory latency. In GPUs, the issue is bypassed on architecture level. They enjoy a high degree of concurrency; GPU cores simply switch to another thread instead of waiting. The CPU cores only run 2 threads each due to hyper-threading, not enough threads to efficiently hide memory latency the way GPUs are doing.

dragontamer 3y ago

My untested rule of thumb is that if something fits inside of GPU-register space, its probably faster on GPU. (GPUs have more architectural registers than CPUs, since CPUs are using all those "shadow registers" to implement out-of-order stuffs). Compilers will automatically turn your most-used variables into registers today, though you might need to verify by looking at the assembly code whether or not you're actually in register-space. But if something fits inside of CPU-cache space, its probably faster on CPU. (Intel is upto 2MB L2 cache on some systems, AMD is up to 192MB L3 cache on some systems). But if something is outside of CPU-cache space but fits inside of GPU-VRAM space, its probably faster on GPU. (Better to be bandwidth-limited at 500GB/s GPU-VRAM speeds than 50GB/s DDR5 speed). Ex: 16GBs GPU-VRAM is back into GPU-sized solutions to a problem. Then if something fits inside of CPU-RAM space, which is like 2TB in practice (!!!!), then CPUs are probably faster. Because there's no point taking a 2TB DDR5 RAM limited problem, passing some of that data to 16GB or 96GB of GPU-VRAM, and then waiting for DDR5 RAM anyway. ------------ Not that I've tested out this theory. But it'd make sense in my brain at least, lol.

deleted 3y ago
[deleted]

Const-me 3y ago

OP has tested the 4600G APU. Theoretically, the chip can do 710 FP32 GFlops on CPU, and 1702 GFlops on the integrated GPU. Another thing, it’s hard to write CPU code which saturates memory throughput instead of stalling on memory latency. In GPUs, the issue is bypassed on architecture level. They enjoy a high degree of concurrency; GPU cores simply switch to another thread instead of waiting. The CPU cores only run 2 threads each due to hyper-threading, not enough threads to efficiently hide memory latency the way GPUs are doing.