One interesting thing is if the model can’t fit into GPU memory (sharded across multiple chips) it would be much slower. So one chip would take more than a mont

One interesting thing is if the model can’t fit into GPU memory (sharded across multiple chips) it would be much slower. So one chip would take more than a month. That’s why these clusters are so valuable, even with Nvidias margins they are still cheaper than using less compute for longer.