Yeah I think you couldn't train GPT-3 on this. You need something incredibly compute and memory bandwidth intensive, but with relatively few parameters. But the

Yeah I think you couldn't train GPT-3 on this. You need something incredibly compute and memory bandwidth intensive, but with relatively few parameters. But there ought to be some interesting architectures to discover in that direction. If nothing else you could do crazy fast hyperparameter optimization on smaller architectures and get much better performance out of them. The architectures we have are shaped by the hardware they run on, I think more than people realize. Different hardware will need different architectures to be fully utilized. Unfortunately there are so few of these chips in existence that it will be hard to get enough people using them to really discover how to best use them.