3 ms·

I've worked in this space for the past five years. The chips are essentially highly parallel processors. There's no unifying architecture. You have the graph-ba

I've worked in this space for the past five years. The chips are essentially highly parallel processors. There's no unifying architecture. You have the graph-based / hpc-simulator chips like Cerebras, Graphcore, etc which are basically a stick-as-many-cores-as-possible situation with a high-speed networking fabric. You have the 'tensor' cores like Groq where the chip operates as a whole and is just well suited for tensor processing (parallelizable, high-speed memory, etc). At the end of the day, it's matrix multiplication acceleration mostly, and then IO optimization. Literally most of the optimization has nothing to do with compute. We can compute faster than we can ingest.

rramadass 3y ago
>The chips are essentially highly parallel processors Right. AFAIK we already were doing SIMD, Vector Processing, VLIW etc. to speed up parallel processing for numerical calculations in AI/ML. What then is the reason for the explosion of these different categories of chips? Are they just glorified ASICs designed for their specific domains or FPGAs programmed accordingly? If so what is their architecture i.e. what are their functional units and how do they work together?
- anon291 3y ago
  Realistically, a good AI chip will have provisions for high throughput IO (the most important thing, and the key differentiator) and the actual processing really doesn't matter because with enough engineering effort you'll be able to saturate the chip. GPUs have a high speed output in the form of an HDMI link. However, there is no high speed input. Reads/writes to/from the GPU are slow. The Cerebras wafer chip for example has 8-16 FPGA driven IO chips that directly read from TCP/IP onto the chip and off again in parallel. Each FPGA connects to its own ethernet port. So you can get the data on/off the chip as fast as possible. That's it really. As for the processing engines. They're usually just standard cores with a high speed interconnect and maybe some matrix multiplication optimizations. Some, like Groq, have a high speed fabric with specialized processors at various locations.

rramadass 3y ago

>The chips are essentially highly parallel processors Right. AFAIK we already were doing SIMD, Vector Processing, VLIW etc. to speed up parallel processing for numerical calculations in AI/ML. What then is the reason for the explosion of these different categories of chips? Are they just glorified ASICs designed for their specific domains or FPGAs programmed accordingly? If so what is their architecture i.e. what are their functional units and how do they work together?

anon291 3y ago
Realistically, a good AI chip will have provisions for high throughput IO (the most important thing, and the key differentiator) and the actual processing really doesn't matter because with enough engineering effort you'll be able to saturate the chip. GPUs have a high speed output in the form of an HDMI link. However, there is no high speed input. Reads/writes to/from the GPU are slow. The Cerebras wafer chip for example has 8-16 FPGA driven IO chips that directly read from TCP/IP onto the chip and off again in parallel. Each FPGA connects to its own ethernet port. So you can get the data on/off the chip as fast as possible. That's it really. As for the processing engines. They're usually just standard cores with a high speed interconnect and maybe some matrix multiplication optimizations. Some, like Groq, have a high speed fabric with specialized processors at various locations.