We quantize models and get nearly the same performance and we keep seeing new 'small' models performing on par with older larger models. Additionally, people do

We quantize models and get nearly the same performance and we keep seeing new 'small' models performing on par with older larger models. Additionally, people don't need nearly the amount of data that models do in order to learn the same tasks. This implies a lot about how fast it may be possible to learn and how far models currently are from that optimum. Because of this I think the authors conclusions about scaling data up really only apply to current models and training techniques. Based on how fast people can learn, I believe we are likely missing a few things that will allow models to learn vastly faster and deeper with fewer total parameters. My own work, as an example, (github jmward01/lmplay) may show how simple changes to embedding training may make drastic improvements to model training.