> I've seen very compelling results in small scale experiments. Small scale experiments are approximately meaningless in ML. A lot of the clever tricks that im

> I've seen very compelling results in small scale experiments. Small scale experiments are approximately meaningless in ML. A lot of the clever tricks that improve results at smaller scales have their gains BTFO'd by the scaling you give up as you increase the size of the model. The key insight from deep learning was that the gains from doing a smarter thing were dwarfed by doing a dumber thing a lot more. For a smarter trick to be valuable, it needs to not only improve results at the small scale, it needs to not become a bottleneck in your system as you jack up the size.