> Thus to generate a 100 words a second requires moving the model 100 times per second – requiring vast amounts of memory bandwidth. It's actually worse for th

> Thus to generate a 100 words a second requires moving the model 100 times per second – requiring vast amounts of memory bandwidth. It's actually worse for the majority of GPU implementations for large models. The matrices don't fit shared memory so the model is loaded many, many times to shared memory (as tiles). Also, unless you are using Hopper distributed shared memory, CTAs can't even share across them. It would be nice to see a Cerebras solution for pre-training and fine-tuning.