4 ms·
After the failure of FB's Galactica model, I wonder whether LLM are starting to hit diminishing return. I've read of several ways diminishing returns could happ
by joe_the_user 4y ago
After the failure of FB's Galactica model, I wonder whether LLM are starting to hit diminishing return. I've read of several ways diminishing returns could happen:
* Not enough training data - you've used up the Internet (even a percentage of the Internet might be as much as is usable by clever brute force).
* Not enough compute time to fully train (we're not close to that)
* The model covers such a large area that testing is impossible
One thing I'd speculate about is perhaps the more different subjects the program is expected to combine, the more it learns to spout plausible bullshit and clever quips, since for clever humans, that how they relate to stuff they don't know. So "pretentious but uninformed" might be a sign.
- TaylorAlexander 4y agoI still need to read past the first page, but I was reminded this morning of the 2021 paper “On the dangers of stochastic parrots: can language models be too big?” which discusses the harms and diminishing returns of moving to ever larger language models. I had to rush out the door today after seeing this paper come up so I can’t speak much to its content right now. But if anyone wants to read it and reflect here I’d like to hear it. https://dl.acm.org/doi/10.1145/3442188.3445922 https://dl.acm.org/doi/10.1145/3442188.3445922
- version_five 4y agoIt's also likely we're just seeing the limits of what the tech can do. It's not actually "intelligent" it's effectively a cool trick, and at some point the gap between what the trick can do and what a "perfect" AI can do becomes obvious. It's really not that different from Eliza being cool for a few minutes before being obviously found wanting. It just takes longer for the shine to wear off as we get to a better technology. Ditto for self driving cars - a steep climb to a 90% solution, then an impossible chasm to cross to something that actually handles the edge cases.
- klodolph 4y agoIt’s also not trained the way humans are. We’re trained interactively, and from an early age, say things in order to express ourselves and try to get what we want. Machine learning models are generally trained passively.
- dwringer 4y agoThe chatbots using the model over at character.ai give you a sort of genetic algorithmic interface by letting you choose from alternative responses and vote on them as conversations progress, and you can make your own bots then train them and get some very interesting results. I am personally extremely optimistic about the future of such technology and I think we've only begun to scratch the surface collectively.
- ImprobableTruth 4y agoI refuse to believe that anybody who compares LLMs to Eliza has interacted with the technology for more than 10 minutes.
- nmfisher 4y agoI think the poster's point was that initial reactions to LLMs are similar to initial reactions to Eliza. Everyone was gobsmacked for a while, but then once the shine wore off and its limitations were exposed, everyone got bored and moved on quickly.
- cercatrova 4y agoAh yes, the AI Effect in action. https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- lairv 4y agoI am eagerly awaiting GPT4, it might be the case that language models will hit a plateau, just like video games graphics hit a plateau too, with each generation being less and less impressive compared to the previous, while still not being perfect
- dougmwne 4y agoThat’s an odd comparison since unreal engine 4 is incredibly close to live action. I watched the recent matrix demo several times in awe.