The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequent

The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequentially. That’s your limit.