Yeah plus these haven’t been trained on multiple epochs and DeepMind just released Chinchilla which implies we don’t need to scale up as much as we have been wr

Yeah plus these haven’t been trained on multiple epochs and DeepMind just released Chinchilla which implies we don’t need to scale up as much as we have been wrt parameters. Also last month Microsoft published a paper demonstrating deep transformers. We may get to expert level performance on these without any major breakthroughs imo.