4 ms·

Processors are not that fast yet, but computers as a whole are. 20 years ago we had Pentium 3 500MHz, 2 FLOPs/cycle, 1 GFlops. GPU of my desktop PC computes 1

Processors are not that fast yet, but computers as a whole are. 20 years ago we had Pentium 3 500MHz, 2 FLOPs/cycle, 1 GFlops. GPU of my desktop PC computes 11 TFlops, that's 4+ orders of magnitude faster. For CPUs alone it's indeed less impressive, $500 for that P3 inflated to $760, the comparable modern CPU is Threadripper 2920X, 16 flops/cycle * 12 cores * 3.5 GHz = 672 GFlops.

rayiner 8y ago
> GPU of my desktop PC computes 11 TFlops, that's 4+ orders of magnitude faster. But they’re much lower quality (less flexible) FLOPs. If you don’t have much data parallelism or heavy conditional logic for each datum, you’re not going to come close to 11 teraflops.
- Const-me 8y ago
  If you don’t have much data parallelism or have heavy conditional logic, you won't come close to 1GFlops on that CPU either. These 2 FLOPs/cycle are for SSE i.e. 4-wide vector math without any conditions or branches. Update: also, for other tasks, GPU flops are of higher quality than CPU. VRAM bandwidth is ridiculous, much higher than system RAM, computations that are bound by RAM bandwidth will be much faster. GPUs mask RAM latency with hardware multithreading, if your code is parallel and needs to access large amount of data randomly, a GPU will saturate these flops, a CPU will sleep most of the time waiting for data to arrive.
  - glangdale 8y ago
    I think it's safe to say that these flops are different. One is not higher quality than the other. It's possible to conjure up examples where one is more fit for purpose (double-precision and irregular tasks on CPU), massive memory access for GPU. Both architectures have different types of cache which also complicates matters. An example that tortures the CPU (by being 'not much data parallelism or heavy conditional logic') will torture the GPU much worse, but this example will likely not really be "flop" dependent in any case.
    - Const-me 8y ago
      They're different but the differences are quite complex. CPUs can be faster for RAM bandwidth-bound code, if there's not that much data and it fits in caches at least L3, CPU cache bandwidth is very good. GPUs can be faster for heavily conditioned code. If it's the condition on RAM value, a CPU will stall or mispredict branches, both slow, a GPU will switch to another hardware thread which is very fast. Despite these differences, I find flops quite useful in practice to compare performance of different computers across architectures of generations. Computers compute, flops measure how fast they do.
  - rayiner 8y ago
    > These 2 FLOPs/cycle are for SSE i.e. 4-wide vector math without any conditions or branches. Using double precision SSE scaler ops, the Pentium III could execute one addition and one multiplication per cycle. (The throughput was the same with vector math because the Pentium III only had 64-bit SSE units. So a 128-bit packed multiply and a 128-bit packed add, four double precision operations, executed over two cycles.)
- imtringued 8y ago
  Fortunately the most resource demanding algorithms involving grapgics or machine learning can run on GPUs.
quickben 8y ago
That comparison can not be directly made as the 'O' in "FLOPS" is different.
- Const-me 8y ago
  They're math operations on small vectors of 32-bit floating point numbers, they produce same result given same input data. The only difference is that CPU computes stuff on 4-wide vectors of these numbers (modern CPUs up to 16), that GPU on 32-wide vectors (other modern GPUs up to 64).
  - Dylan16807 8y ago
    A lot of the FLOPS on newer CPUs are like GPUs, yes. But the comparison is against an old CPU, without big vector units. Those non-vector calculations are very different, and far more flexible. A brand new CPU core can do 3-4x as many separate operations per cycle, and is clocked 6-10x as fast. Having more cores helps but it's still far behind. Also for a fair price/performance ratio you probably want to compare to the 450MHz model at $230, so only a $350 CPU today ($280 equivalent by august). https://money.cnn.com/1999/08/23/technology/intel/ https://money.cnn.com/1999/08/23/technology/intel/
    - Const-me 8y ago
      > and far more flexible They’re both very flexible, just different. For example, GPUs are better at vectorized condition code. CPUs were mostly fixed with AVX512 but these instructions are too new, only available on some servers. Sure, there’re algorithms which don’t work on GPUs. A stream cipher would be very slow because requires single-thread performance, also GPUs don’t have AES hardware. A compiler is borderline impossible because inherently scalar and requires dynamic memory. Also GPUs don’t do double-precision math particularly fast. Still, I think many users who need high-performance computing can utilize GPUs. They’re trickier to program, but this might be fixable with tools/languages, we have been programming classic computers for ~70 years, GPGPUs for just 12.
      - 8y ago

rayiner 8y ago

> GPU of my desktop PC computes 11 TFlops, that's 4+ orders of magnitude faster. But they’re much lower quality (less flexible) FLOPs. If you don’t have much data parallelism or heavy conditional logic for each datum, you’re not going to come close to 11 teraflops.

Const-me 8y ago
If you don’t have much data parallelism or have heavy conditional logic, you won't come close to 1GFlops on that CPU either. These 2 FLOPs/cycle are for SSE i.e. 4-wide vector math without any conditions or branches. Update: also, for other tasks, GPU flops are of higher quality than CPU. VRAM bandwidth is ridiculous, much higher than system RAM, computations that are bound by RAM bandwidth will be much faster. GPUs mask RAM latency with hardware multithreading, if your code is parallel and needs to access large amount of data randomly, a GPU will saturate these flops, a CPU will sleep most of the time waiting for data to arrive.
- glangdale 8y ago
  I think it's safe to say that these flops are different. One is not higher quality than the other. It's possible to conjure up examples where one is more fit for purpose (double-precision and irregular tasks on CPU), massive memory access for GPU. Both architectures have different types of cache which also complicates matters. An example that tortures the CPU (by being 'not much data parallelism or heavy conditional logic') will torture the GPU much worse, but this example will likely not really be "flop" dependent in any case.
  - Const-me 8y ago
    They're different but the differences are quite complex. CPUs can be faster for RAM bandwidth-bound code, if there's not that much data and it fits in caches at least L3, CPU cache bandwidth is very good. GPUs can be faster for heavily conditioned code. If it's the condition on RAM value, a CPU will stall or mispredict branches, both slow, a GPU will switch to another hardware thread which is very fast. Despite these differences, I find flops quite useful in practice to compare performance of different computers across architectures of generations. Computers compute, flops measure how fast they do.
- rayiner 8y ago
  > These 2 FLOPs/cycle are for SSE i.e. 4-wide vector math without any conditions or branches. Using double precision SSE scaler ops, the Pentium III could execute one addition and one multiplication per cycle. (The throughput was the same with vector math because the Pentium III only had 64-bit SSE units. So a 128-bit packed multiply and a 128-bit packed add, four double precision operations, executed over two cycles.)
imtringued 8y ago
Fortunately the most resource demanding algorithms involving grapgics or machine learning can run on GPUs.

quickben 8y ago

That comparison can not be directly made as the 'O' in "FLOPS" is different.

Const-me 8y ago
They're math operations on small vectors of 32-bit floating point numbers, they produce same result given same input data. The only difference is that CPU computes stuff on 4-wide vectors of these numbers (modern CPUs up to 16), that GPU on 32-wide vectors (other modern GPUs up to 64).
- Dylan16807 8y ago
  A lot of the FLOPS on newer CPUs are like GPUs, yes. But the comparison is against an old CPU, without big vector units. Those non-vector calculations are very different, and far more flexible. A brand new CPU core can do 3-4x as many separate operations per cycle, and is clocked 6-10x as fast. Having more cores helps but it's still far behind. Also for a fair price/performance ratio you probably want to compare to the 450MHz model at $230, so only a $350 CPU today ($280 equivalent by august). https://money.cnn.com/1999/08/23/technology/intel/ https://money.cnn.com/1999/08/23/technology/intel/
  - Const-me 8y ago
    > and far more flexible They’re both very flexible, just different. For example, GPUs are better at vectorized condition code. CPUs were mostly fixed with AVX512 but these instructions are too new, only available on some servers. Sure, there’re algorithms which don’t work on GPUs. A stream cipher would be very slow because requires single-thread performance, also GPUs don’t have AES hardware. A compiler is borderline impossible because inherently scalar and requires dynamic memory. Also GPUs don’t do double-precision math particularly fast. Still, I think many users who need high-performance computing can utilize GPUs. They’re trickier to program, but this might be fixable with tools/languages, we have been programming classic computers for ~70 years, GPGPUs for just 12.
    - 8y ago