3 ms·

Since people seem to be wondering how the architecture works, and the software stack is open (although I'm not sure to what extend), I'll share my understanding

Since people seem to be wondering how the architecture works, and the software stack is open (although I'm not sure to what extend), I'll share my understanding of it. The basic system consists of the cards consists of a bunch of Tensix cores and shared memory: > Each Tensix core contains a high-density tensor math unit (FPU) which performs most of the heavy lifting, a SIMD engine (SFPU), five Risc-V CPU cores, and a large local memory storage. [0] > The cores are connected with two torus-shaped, going in opposite directions. [0] The RISC-V cores in the Tensix cores, are tiny rv32i cores, that can control the FPU, SFPU, and are also used to prepare/move data. The FPU, does "dense tensor math", so I think it's probably a matmul engine, but I don't know any more specifics. [1] The SFPU is a more general purpose SIMT engine, that can be driven from the RISC-V cores. There is a SFPU simulator you can play around with on their github. [2] See the low level Kernels example for how the programming model works. [3] The grayskull SFPU has 4 general purpose LRegs, which each hold 64 19-bit values. Wormhole has 8 general purpose LRegs, which each hold 32 32-bit values. I've been told that wormhole SFPU has a ~3x IPC increase over grayskull, and a few new SFPU instructions. You can probably find out more by browsing the docs and rummaging through the github repos. [4] [0] https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware [1] https://docs.tenstorrent.com/tenstorrent/v/tt-buda/terminology#fpu https://docs.tenstorrent.com/tenstorrent/v/tt-buda/terminolo... [2] https://github.com/tenstorrent-metal/sfpi/blob/master/tests/grayskull/sfpi/sfpu.cc https://github.com/tenstorrent-metal/sfpi/blob/master/tests/... [3] https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal/apis/kernel_apis/sfpu/llk.html https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal... [4] https://github.com/tenstorrent-metal/ https://github.com/tenstorrent-metal/

BirAdam 3y ago
So, modern transputer but RISC-V. Seems neat.
cmrdporcupine 3y ago
Oh, so am I to read this right that the actual vector work being done here is not done via the RiscV [V]ector extension, but by a purely custom core?
- camel-cdr 3y ago
  Yes, the RISC-V Vector extension would be absolutely overkill for an ML accelerator, and waste a lot of die space. Currently the cards need an x86_64 host, but they plan to replace that with their ascalon risc-v cpu, that supports the risc-v vector extension (rvv). So most compute is done by the accelerator cards, but for some things the host system steps in. There rvv can come in handy, because it's a lot more flexible.
  - cmrdporcupine 3y ago
    It sounds very cool. I've been following them since watching a couple interviews with Jim Keller, and then seeing they're (in part) a Toronto area local company. I don't work in machine-learning but find the HW direction really interesting. I could think of a few uses for that kind of high bandwidth vector/matrix crunching that aren't restricted to ML purposes.

BirAdam 3y ago

So, modern transputer but RISC-V. Seems neat.

cmrdporcupine 3y ago

Oh, so am I to read this right that the actual vector work being done here is not done via the RiscV [V]ector extension, but by a purely custom core?

camel-cdr 3y ago
Yes, the RISC-V Vector extension would be absolutely overkill for an ML accelerator, and waste a lot of die space. Currently the cards need an x86_64 host, but they plan to replace that with their ascalon risc-v cpu, that supports the risc-v vector extension (rvv). So most compute is done by the accelerator cards, but for some things the host system steps in. There rvv can come in handy, because it's a lot more flexible.
- cmrdporcupine 3y ago
  It sounds very cool. I've been following them since watching a couple interviews with Jim Keller, and then seeing they're (in part) a Toronto area local company. I don't work in machine-learning but find the HW direction really interesting. I could think of a few uses for that kind of high bandwidth vector/matrix crunching that aren't restricted to ML purposes.