4 ms·
> There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the
by cma 2y ago
> There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order..
GPT4 used the tensor programs (Greg Yang,now at X. Ai) research to effectively transfer hyper parameters from smaller models where experiments can go fast to the larger one in a predictable way and got a very smooth loss curve throughout the training.
- noosphr 2y agoBut that's OPs point. You need to care about those hyper parameters. Good on them for finding them in smaller models, but they still had to find them, because without those hyper parameters pre-training is at best unstable and at worst divergent. Also small here means hundreds of millions to billions of parameters - that's still larger than any model pre 2020.
- cma 2y agoPrior to that work on hyperparameter transfer though things would go wrong much more often training the big expensive model and require restarts back to checkpoints and retuning parameters much more often though. Yeah it's still a lot of fiddling around but it is orders of magnitude less expensive. But that work also went far beyond transfer of fiddling around and mathematically solved a lot of stuff with avoiding overfitting, great video overview of it here: https://www.youtube.com/watch?v=1aXOXHA7Jcw https://www.youtube.com/watch?v=1aXOXHA7Jcw