If you have a computation that can be expressed as a definite and finite sequence of steps, such as computing the next token from an LLM, you could unwind all t

If you have a computation that can be expressed as a definite and finite sequence of steps, such as computing the next token from an LLM, you could unwind all the loops and spread it amongst a grid of FPGAs. It would be large, expensive, and power hungry, but doable. If you do that, you're likely to have a latency on the order of almost a millisecond, putting the previous tokens in one end would get you the logits for the next at a rate of let's say 1000 tokens per second... impressive at current rates. You could also take that same array, and program in several latches along the way to synchronize data at selected points, and enabling pipelining. This might produce a slight (10%) increase in latency, so a 10% or so loss in throughput for a single stream. However, it would allow you to have multiple independent streams flowing through the FPGAs. Instead of serving 1 customer at 1000 tokens/second, you might have 10 or more customers each with their 900 tokens/second. Parallelism and pipelining are the future of compute.