4 ms·
Beautiful website and nice fonts! but back to the topic: I’m quite shocked that PaLM gets outclassed by much smaller models on a regular basis. I would have th
by behnamoh 3y ago
Beautiful website and nice fonts!
but back to the topic: I’m quite shocked that PaLM gets outclassed by much smaller models on a regular basis. I would have thought that Google, despite not having a moat, at least had enough talent and focus to get LLMs right. But what I’m observing is that startups like ClosedAI, Anthrophic, etc. constantly beat big players like Google in their own game.
- solrik 3y agoI am sure Google is working on some great new models. PalM was a disappointment, but they have been leading the charge in deep learning for decades.
- cubefox 3y agoThey (with the new combined Google DeepMind team) do indeed work on a new large model: Gemini. The intention seems to be to outperform GPT-4. During Google IO a while ago, Pichai said Gemini was currently in training.
- cubefox 3y agoNote that PaLM and PaLM 2 are completely different models.
- theage 3y agoLook at Hollywood, they spawned photo-realistic CGI and figured out their captive audience will settle for much lesser quality anyway so why bother.
- CyberDildonics 3y agoHollywood is a city. If you mean movies, every production is different and has its own budget. They don't have a captive audience, people have to choose to watch a movie for entertainment and then choose a specific movie. I don't know what you mean by 'much lesser' quality but hundreds of millions in CG were put into just the biggest movies of the summer. Avatar 2 alone was an enormous feat. No one at any point in the process is capable of making something spectacular, then deciding to just make something that looks mediocre instead. The only place that happens is cartoons for kids.
- gwern 3y agoWhat happened was Google beat itself at its own game, if you will. Google PaLM-1 is beaten so often simply because Google Chinchilla scaling is so much better. (Note that PaLM-2 is not benchmarked in OP.) PaLM-1 is hobbled by the fact that it was probably the largest (because the last) LLM to be trained with the Kaplan scaling laws rather than the Chinchilla. As soon as Chinchilla came out, no one would train like PaLM-1 again, because it was giving up so much performance compared to if one had instead trained a much smaller Chinchilla-optimal model. (This had the interesting consequence that PaLM-1 would thereby remain the largest, by parameter-count, dense LLM trained for probably years to come - because why would you train one inefficiently as that, while a larger-than-PaLM-1 Chinchilla-optimal model would require staggering levels of compute+data.) The PaLM-1 paper came out within days of the Chinchilla paper, and many people noted that this pointed to extraordinary levels of dysfunctionality within Google - that DeepMind would not tell Google Brain that they were wasting literally millions of dollars of compute by training a model in what DeepMind was busy showing was a very suboptimal way.
- behnamoh 3y agothanks for your explanation, this clarified a lot of things.
- flkenosad 3y agoSo could Google now just spend another few millions of compute training with Chinchilla and be exponentially further along?
- gwern 3y agoYou mean of PaLM-1? No. PaLM-1 was extremely far off Chinchilla scaling, it's not something you can salvage with just a few more millions of TPU-pod time. Think more like, hundreds of millions... I think somewhere I estimated PaLM-1 was at something like 5% of Chinchilla-optimal, so they would have to train it 20x more. Obviously, they are not going to, not with methods improving so rapidly, with innovations like UL2. PaLM-1 is just a sunk cost, is all, and a useful historical datapoint (eg. studying inverse scaling). Equally obviously, PaLM-2 did not make the same mistake and so for that and other reasons, greatly outperforms PaLM-1. So, beating PaLM-1 at this point is an achievement, but you are still far from 'beating Google'.