3 ms·

Is there any quick summary about how exactly (architecturally) these LPUs differ from GPUs or TPUs? Intuitively where does the speedup come from, or what tradeo

Is there any quick summary about how exactly (architecturally) these LPUs differ from GPUs or TPUs? Intuitively where does the speedup come from, or what tradeoffs are being made in order to achieve them?

anon291 3y ago
There's a youtube video somewhere. Intuitively, the speedup comes from the fact that the pipelining is controlled by software, rather than hardware. There's no memory management unit. Memory operations and timing is controlled by software entirely. This is terrible for CPUs, where it's really important to do register renaming, out-of-order execution, etc. However, for AI models, it doesn't really matter. Tensor accesses are easily optimized by software, which knows exactly how the chip runs. Essentially, the chip is able to read memory, write memory, do vector operations, multiply matrices, and do permutations all in the same cycle, and in fact it can have several of these operations going on at the same time. The compiler obviously knows how to schedule all this to produce good timing. That's why the utilization is so high. Every piece of the chip is being used. Whereas with a GPU model, you have to kind of play 'code golf' and have a deep understanding of the architecture and the execution engine to be able to determine when a memory access is going to cause a pipeline stall. If you understand the ISA of the GroqChip, you understand completely how the chip works. There is no pipeline stall. There is no waiting for memory. There is no memory hierarchy. Even the interconnects work at the same speed as the chip (or something like that... it's all in the paper, so public information), so when they network chips together, the latency between any two subunits of every chip is already known by the network topology and the chip timings. TL;DR It's all very deterministic and this works well for tensor operations. Parallelism is determined at compile time Source: https://www.youtube.com/watch?v=pb0PYhLk9r8 https://www.youtube.com/watch?v=pb0PYhLk9r8 . This is basically exactly how the chip works. It's not some simplified architectural or marketing diagram. It's literally how it works.
- kseniase 3y ago
  I re-posted your comment here: https://www.turingpost.com/p/fod41 https://www.turingpost.com/p/fod41
- sifar 3y ago
  >> There is no pipeline stall. There is no waiting for memory. I mean this is a bit of a hyperbole. Even if things are deterministic, the compiler might not be able to schedule things optimally, their can be pipeline interlocks or memory bank conflicts (depending on sizes involved) in some scenarios.
  - anon291 3y ago
    There is no implicit pipeline stalls. Any pipeline stalls are detectable at compile time via trivial static analysis tools, letting engineers work around it, or decide it's fine. Sorry if that was confusing.

anon291 3y ago

There's a youtube video somewhere. Intuitively, the speedup comes from the fact that the pipelining is controlled by software, rather than hardware. There's no memory management unit. Memory operations and timing is controlled by software entirely. This is terrible for CPUs, where it's really important to do register renaming, out-of-order execution, etc. However, for AI models, it doesn't really matter. Tensor accesses are easily optimized by software, which knows exactly how the chip runs. Essentially, the chip is able to read memory, write memory, do vector operations, multiply matrices, and do permutations all in the same cycle, and in fact it can have several of these operations going on at the same time. The compiler obviously knows how to schedule all this to produce good timing. That's why the utilization is so high. Every piece of the chip is being used. Whereas with a GPU model, you have to kind of play 'code golf' and have a deep understanding of the architecture and the execution engine to be able to determine when a memory access is going to cause a pipeline stall. If you understand the ISA of the GroqChip, you understand completely how the chip works. There is no pipeline stall. There is no waiting for memory. There is no memory hierarchy. Even the interconnects work at the same speed as the chip (or something like that... it's all in the paper, so public information), so when they network chips together, the latency between any two subunits of every chip is already known by the network topology and the chip timings. TL;DR It's all very deterministic and this works well for tensor operations. Parallelism is determined at compile time Source: https://www.youtube.com/watch?v=pb0PYhLk9r8 https://www.youtube.com/watch?v=pb0PYhLk9r8 . This is basically exactly how the chip works. It's not some simplified architectural or marketing diagram. It's literally how it works.

kseniase 3y ago
I re-posted your comment here: https://www.turingpost.com/p/fod41 https://www.turingpost.com/p/fod41
sifar 3y ago
>> There is no pipeline stall. There is no waiting for memory. I mean this is a bit of a hyperbole. Even if things are deterministic, the compiler might not be able to schedule things optimally, their can be pipeline interlocks or memory bank conflicts (depending on sizes involved) in some scenarios.
- anon291 3y ago
  There is no implicit pipeline stalls. Any pipeline stalls are detectable at compile time via trivial static analysis tools, letting engineers work around it, or decide it's fine. Sorry if that was confusing.