3 ms·

Is this something from a research finding or is it your idea?

mikewarot 3y ago
It's my idea... I've been writing about it for over a decade. I hope to get a small version of it made via tiny tapeout later this year once I'm out of the precariat. The idea is that you can decompose any non-cyclic code, for example, a matrix multiply, FFT, etc.. into a directed graph of bitwise operations, then map those operations out into the grid of LUTs. Pipe inputs in one side of the grid, and get the outputs out the other side, and all of the bitwise operations happen in lock step parallelism. (Unlike an FPGA which is asyncronous) If you can decompose one stage of an LLM down into a bitwise graph of computation, you can easily map it into the grid. If there's a bad cell in the grid, you can map around it. Because all of the high speed logic lines are short (to the neighbors) you don't have the long lines / high capacitance issues driving signals all the way across an FPGA or ASIC, thus you can really crank up the clock rate (and/or it's very power efficient). It should be trivial, design wise, to scale from a 16x16 grid up through chips that could crank out Petaflops. Here's a simulator for the thing, written in Pascal. https://github.com/mikewarot/Bitgrid https://github.com/mikewarot/Bitgrid
- sitkack 3y ago
  You can simulate this on a GPU now. This is like running a CA on a GPU with slightly different rules. I think you would also be delighted to know that many of the non-GPU ai accelerators are in a mesh topology. Cerebras, Tenstorrent, Esperanto https://www.esperanto.ai/wp-content/uploads/2021/08/HC2021.Esperanto.Ditzel.Final_.pdf https://www.esperanto.ai/wp-content/uploads/2021/08/HC2021.E...
  - mikewarot 3y ago
    I've been intrigued by this approach. On a more highly optimized (but harder to program) take is the GA144[1] from Chuck Moore, the inventor of Forth. It's a grid of 144 F18 Forth based processors in a cartesian grid. These processors are far more limited, but then again they take far less power as well. [1] https://www.greenarraychips.com/ https://www.greenarraychips.com/
    - sitkack 3y ago
      That is a pretty cool chip. Chuck Moore is a fascinating character. You would probably get a kick out of David Ackley's T2 Tile Project. https://t2tile.com/ https://t2tile.com/ https://www.youtube.com/@T2TileProject/videos https://www.youtube.com/@T2TileProject/videos https://www.youtube.com/@DaveAckley https://www.youtube.com/@DaveAckley https://en.wikipedia.org/wiki/Ackley_function https://en.wikipedia.org/wiki/Ackley_function https://www.cs.unm.edu/~ackley/ https://www.cs.unm.edu/~ackley/ Fun fact, Tenstorrent wanted to add instructions to enqueue data between processors connected in a mesh and Arm said no (they don't do architectural licenses anymore), so Tenstorrent used RISCV. Implementing something like a GA144 but using small RISC-V processors would be an interesting hack. https://www.youtube.com/watch?v=6QRKpd28NEE https://www.youtube.com/watch?v=6QRKpd28NEE

It's my idea... I've been writing about it for over a decade. I hope to get a small version of it made via tiny tapeout later this year once I'm out of the precariat. The idea is that you can decompose any non-cyclic code, for example, a matrix multiply, FFT, etc.. into a directed graph of bitwise operations, then map those operations out into the grid of LUTs. Pipe inputs in one side of the grid, and get the outputs out the other side, and all of the bitwise operations happen in lock step parallelism. (Unlike an FPGA which is asyncronous) If you can decompose one stage of an LLM down into a bitwise graph of computation, you can easily map it into the grid. If there's a bad cell in the grid, you can map around it. Because all of the high speed logic lines are short (to the neighbors) you don't have the long lines / high capacitance issues driving signals all the way across an FPGA or ASIC, thus you can really crank up the clock rate (and/or it's very power efficient). It should be trivial, design wise, to scale from a 16x16 grid up through chips that could crank out Petaflops. Here's a simulator for the thing, written in Pascal. https://github.com/mikewarot/Bitgrid https://github.com/mikewarot/Bitgrid

sitkack 3y ago
You can simulate this on a GPU now. This is like running a CA on a GPU with slightly different rules. I think you would also be delighted to know that many of the non-GPU ai accelerators are in a mesh topology. Cerebras, Tenstorrent, Esperanto https://www.esperanto.ai/wp-content/uploads/2021/08/HC2021.Esperanto.Ditzel.Final_.pdf https://www.esperanto.ai/wp-content/uploads/2021/08/HC2021.E...
- mikewarot 3y ago
  I've been intrigued by this approach. On a more highly optimized (but harder to program) take is the GA144[1] from Chuck Moore, the inventor of Forth. It's a grid of 144 F18 Forth based processors in a cartesian grid. These processors are far more limited, but then again they take far less power as well. [1] https://www.greenarraychips.com/ https://www.greenarraychips.com/
  - sitkack 3y ago
    That is a pretty cool chip. Chuck Moore is a fascinating character. You would probably get a kick out of David Ackley's T2 Tile Project. https://t2tile.com/ https://t2tile.com/ https://www.youtube.com/@T2TileProject/videos https://www.youtube.com/@T2TileProject/videos https://www.youtube.com/@DaveAckley https://www.youtube.com/@DaveAckley https://en.wikipedia.org/wiki/Ackley_function https://en.wikipedia.org/wiki/Ackley_function https://www.cs.unm.edu/~ackley/ https://www.cs.unm.edu/~ackley/ Fun fact, Tenstorrent wanted to add instructions to enqueue data between processors connected in a mesh and Arm said no (they don't do architectural licenses anymore), so Tenstorrent used RISCV. Implementing something like a GA144 but using small RISC-V processors would be an interesting hack. https://www.youtube.com/watch?v=6QRKpd28NEE https://www.youtube.com/watch?v=6QRKpd28NEE