3 ms·

But that's OPs point. You need to care about those hyper parameters. Good on them for finding them in smaller models, but they still had to find them, because w

But that's OPs point. You need to care about those hyper parameters. Good on them for finding them in smaller models, but they still had to find them, because without those hyper parameters pre-training is at best unstable and at worst divergent. Also small here means hundreds of millions to billions of parameters - that's still larger than any model pre 2020.

cma 2y ago
Prior to that work on hyperparameter transfer though things would go wrong much more often training the big expensive model and require restarts back to checkpoints and retuning parameters much more often though. Yeah it's still a lot of fiddling around but it is orders of magnitude less expensive. But that work also went far beyond transfer of fiddling around and mathematically solved a lot of stuff with avoiding overfitting, great video overview of it here: https://www.youtube.com/watch?v=1aXOXHA7Jcw https://www.youtube.com/watch?v=1aXOXHA7Jcw