268 ms·
They did not publish benchmarks about quality of the models, which is very suspicious. I personally squinted hard when they said removing dropout improves trai
by lostmsu 4y ago
They did not publish benchmarks about quality of the models, which is very suspicious.
I personally squinted hard when they said removing dropout improves training speed (which is in iterations per second), but said nothing about how it affects the performance (rate of mistakes in inference) of the trained model.
- jasonphang 4y agoI agree that the lack of benchmarks makes it hard to determine how valuable this model is. But on the topic of dropout, dropout has been dropped for the pretraining stage of several other large models. Off the top of my head: GPT-J-6B, GPT-NeoX-20B, and T5-1.1/LM.