6 ms·
Model performance doesn't seem to be monotonically increasing with size. The 84B parameter model is the best on most tasks. What happened to "you just train it
by civilized 3y ago
Model performance doesn't seem to be monotonically increasing with size. The 84B parameter model is the best on most tasks.
What happened to "you just train it on more data and performance goes up exponentially"? There were so many charts proving this and we could even project at what level of data/compute world-conquering superintelligence would inevitably emerge.
- pptr 3y agoMaybe there is simply not enough medical data to properly train a 500B model.
- isaacfung 3y agoThere is text under the table.
- civilized 3y agoThey didn't try particularly hard to find out why the biggest model didn't perform as well. Or at all, as far as they report. They give us one sentence speculating "maybe not enough training". Come on. This is Google. They have unlimited compute resource, biomedical AI is a core strategic objective, they've spent untold $$$ getting data and working on this for years. There are news articles on their efforts going back a decade. And all they could do is wave their hands at "maybe not enough training".