6 ms·
So could Google now just spend another few millions of compute training with Chinchilla and be exponentially further along?
by flkenosad 3y ago
So could Google now just spend another few millions of compute training with Chinchilla and be exponentially further along?
- gwern 3y agoYou mean of PaLM-1? No. PaLM-1 was extremely far off Chinchilla scaling, it's not something you can salvage with just a few more millions of TPU-pod time. Think more like, hundreds of millions... I think somewhere I estimated PaLM-1 was at something like 5% of Chinchilla-optimal, so they would have to train it 20x more. Obviously, they are not going to, not with methods improving so rapidly, with innovations like UL2. PaLM-1 is just a sunk cost, is all, and a useful historical datapoint (eg. studying inverse scaling). Equally obviously, PaLM-2 did not make the same mistake and so for that and other reasons, greatly outperforms PaLM-1. So, beating PaLM-1 at this point is an achievement, but you are still far from 'beating Google'.