Almost all the challenges with GPT-sized models are engineering and training challenges, not architectural. How do you train a model too big to fit in a single

Almost all the challenges with GPT-sized models are engineering and training challenges, not architectural. How do you train a model too big to fit in a single GPU? It's doable, but not simple. How do you update weights across your cluster? etc etc