4 ms·

I post this in the spirit of being corrected if I am wrong, the Internet's core skill. If I'm reading this correctly, the competition for this technology isn't

by jerf 5y ago

I post this in the spirit of being corrected if I am wrong, the Internet's core skill. If I'm reading this correctly, the competition for this technology isn't CPUs, it's GPUs. If that's the case, it seems like they'd be able to show a lot of artificially inflated numbers vs. CPUs simply by running already-known-GPU-heavy workloads on their hardware vs. CPUs and show massive performance gains vs. the CPUs... but what about an optimal GPU algorithm? Traditionally, making the hardware that can support reconfigurability at a highly granular level (FPGAs, for instance) has never been able to catch up with conventional silicon, because the you can't recover the quite significant costs of all that reconfigurability with improved architecture. The conventional silicon will still beat you with much faster speeds even if you take a penalty vs. some theoretical maximum they could have achieved with some other architecture. Plus, again, GPUs really cut into the whole space of "things CPU can't do well". They haven't claimed the whole space, but they certainly put their marker down on the second-most profitable segment of that space. Between a strong CPU and a strong GPU you've got quite a lot of computing power across a very wide set of possible computation tasks. I am not an expert in this space, but it seems like the only algorithms I've seen lately people proposing custom hardware for at scale are neural network evaluation. By "at scale", I mean, there's always "inner loop" sorts of places where someone has some specific task they need an ASIC for, encryption being one of the classic examples of something that has swung back and forth between being done in CPUs and being done in ASICs for decades now. But those problems aren't really a candidate for this sort of tech because they are so in need of hyperoptimization that they do in fact go straight to ASICs. It doesn't seem to me like there's a huge middle ground here anymore between stuff worth dedicating the effort to make ASICs for (which almost by definition will outperform anything else), and the stuff covered by some combination of CPU & GPU. So... how am I misreading this, oh great and mightily contentious Internet?

pclmulqdq 5y ago
You are correct that the main competing technology is throughput compute (GPUs, TPUs, ML processors) rather than "general-purpose" CPU-style compute with batch size of 1. The way we use CPUs is fairly unique, though, where we time share a lot of different code on each core of a system. Trying to move CPU applications to an FPGA-style device would be a disaster. I used to believe that FPGAs could compete with GPUs in throughput compute cases thanks to high-level synthesis and OpenCL, but it looks like most applications have gone with something more ASIC-like than general-purpose (eg TPUs and matrix units in GPUs for ML).
mhh__ 5y ago
In comparing with CPUs I would also want to see thorough measurement of latency too.
dragontamer 5y ago
CPUs __ARE__ an ASIC. You can get a commodity CPU today at 4.7+ GHz clock and 64 MB of L3 cache (AMD Ryzen 9 5950X). There's no FPGA / configurable logic in the world that comes even close. AMD's even shown off a test chip with 2x96 MB of L3 cache. GPUs __ARE__ an ASIC, but with a different configuration than CPUs. An AMD MI100 comes with 120 compute units, each CU has 4x4x16 SIMD-lanes and each lane has 256 x 32-bit __registers__. (AMD runs each lane once-per-4 clock ticks) That's 32MB of __registers__, accessible in once every 4th clock tick. Sure, a clock of 1.5GHz is much slower but you ain't getting this number of registers or SRAM on any FPGA. > I am not an expert in this space, but it seems like the only algorithms I've seen lately people proposing custom hardware for at scale are neural network evaluation The FPGA / ASIC stuff is almost always "pipelined systolic array". The systolic array is even more parallel than a GPU, and requires data to move "exactly" as planned. The idea is that instead of moving data to / from central registers, you move data across the systolic array to each area that needs it. ---------- CPUs: Today's CPUs __discover__ parallelism in standard assembly code (aka: Instruction level parallelism). It spends a lot of energy "decoding", which is really "scheduling" which instruction should run in which of some 8 to 16 pipelines (depending on AMD Zen vs Skylake vs Apple M1). Each pipeline has different characteristics (this one can add / multiply, but another one can add / multiply / divide). Sometimes some instructions take up the whole pipeline (divide), other times, multiply takes 5 clock ticks but can "take an instruction" every clock tick. CPUs spend a huge amount of effort discovering the optimal, parallel, execution for any chunk of assembly code, and today is scanning ~400 to 700 instructions to do so (the rough size of a typical CPU's reorder buffer). That's why branch prediction is so important. In effect: CPUs are today's ultimate MIMD (multiple instructions / multiple data) computers... with a big decoder in front ensuring the various pipelines remain full. ----------- GPUs: SIMD execution. If parallelism is known ahead of time, this is a better architecture. Instead of spending so much space and energy on discovering parallelism, you have the programmer explicitly write a parallel program from the start. GPUs are the today's ultimate SIMD computers (single instruction / multiple data) computers. ------------- Systolic Array: Will never be a general processor, must be designed for each specific task at hand. Can only work if all data-movements are set in stone ahead of time (such as matrix multiplication). Under these conditions, is even more parallel and efficient than a GPU.
- 5y ago
dw-im-here 5y ago
Yes you are

by jerf 5y ago

pclmulqdq 5y ago

You are correct that the main competing technology is throughput compute (GPUs, TPUs, ML processors) rather than "general-purpose" CPU-style compute with batch size of 1. The way we use CPUs is fairly unique, though, where we time share a lot of different code on each core of a system. Trying to move CPU applications to an FPGA-style device would be a disaster. I used to believe that FPGAs could compete with GPUs in throughput compute cases thanks to high-level synthesis and OpenCL, but it looks like most applications have gone with something more ASIC-like than general-purpose (eg TPUs and matrix units in GPUs for ML).

mhh__ 5y ago

In comparing with CPUs I would also want to see thorough measurement of latency too.

dragontamer 5y ago

CPUs __ARE__ an ASIC. You can get a commodity CPU today at 4.7+ GHz clock and 64 MB of L3 cache (AMD Ryzen 9 5950X). There's no FPGA / configurable logic in the world that comes even close. AMD's even shown off a test chip with 2x96 MB of L3 cache. GPUs __ARE__ an ASIC, but with a different configuration than CPUs. An AMD MI100 comes with 120 compute units, each CU has 4x4x16 SIMD-lanes and each lane has 256 x 32-bit __registers__. (AMD runs each lane once-per-4 clock ticks) That's 32MB of __registers__, accessible in once every 4th clock tick. Sure, a clock of 1.5GHz is much slower but you ain't getting this number of registers or SRAM on any FPGA. > I am not an expert in this space, but it seems like the only algorithms I've seen lately people proposing custom hardware for at scale are neural network evaluation The FPGA / ASIC stuff is almost always "pipelined systolic array". The systolic array is even more parallel than a GPU, and requires data to move "exactly" as planned. The idea is that instead of moving data to / from central registers, you move data across the systolic array to each area that needs it. ---------- CPUs: Today's CPUs __discover__ parallelism in standard assembly code (aka: Instruction level parallelism). It spends a lot of energy "decoding", which is really "scheduling" which instruction should run in which of some 8 to 16 pipelines (depending on AMD Zen vs Skylake vs Apple M1). Each pipeline has different characteristics (this one can add / multiply, but another one can add / multiply / divide). Sometimes some instructions take up the whole pipeline (divide), other times, multiply takes 5 clock ticks but can "take an instruction" every clock tick. CPUs spend a huge amount of effort discovering the optimal, parallel, execution for any chunk of assembly code, and today is scanning ~400 to 700 instructions to do so (the rough size of a typical CPU's reorder buffer). That's why branch prediction is so important. In effect: CPUs are today's ultimate MIMD (multiple instructions / multiple data) computers... with a big decoder in front ensuring the various pipelines remain full. ----------- GPUs: SIMD execution. If parallelism is known ahead of time, this is a better architecture. Instead of spending so much space and energy on discovering parallelism, you have the programmer explicitly write a parallel program from the start. GPUs are the today's ultimate SIMD computers (single instruction / multiple data) computers. ------------- Systolic Array: Will never be a general processor, must be designed for each specific task at hand. Can only work if all data-movements are set in stone ahead of time (such as matrix multiplication). Under these conditions, is even more parallel and efficient than a GPU.

5y ago

dw-im-here 5y ago

Yes you are