Most of the work of LLMs is in large matrix multiply-accumulate operations. You could take all those constants and convert everything to fixed point, then compi

Most of the work of LLMs is in large matrix multiply-accumulate operations. You could take all those constants and convert everything to fixed point, then compile it down to the smallest possible directed acyclic graph of binary logical operations. (In other words, do the NAND to Tetris thing in reverse) The problem is FPGAs are heterogeneous and highly optimized to reduce latency, instead of for efficiency, so there are strong limits to the approach. On the plus side, the LLMs are a lot of layers, so you could take one layer per FPGA, and just use the high speed links in the chips to feed the results to the next FPGA. You'd have maybe even a millisecond of latency, at a clock rate of 100 Mhz, but possibly a million tokens per second of serial / parallel streams of execution.