4 ms·
> for instance, they have broad general knowledge about science, but don’t know much about popular culture That seems like a good focus. Why learn details that
by diggan 1y ago
> for instance, they have broad general knowledge about science, but don’t know much about popular culture
That seems like a good focus. Why learn details that can change within days of it being released? Instead, train the models to have good general knowledge, and be really good at using tools, and you won't have to re-train models from scratch just because some JS library now has a different API, instead the model goes out to fetch the latest APIs/gossip when needed.
- wmf 1y agoYeah, it always seemed like a sad commentary on our world that AIs are devoting their weights to encyclopedic knowledge of Harry Potter, Pokemon, and Reddit trolling.
- eru 1y agoWhy? You gotta provide what your customers want. And it's far from sad that we have so many resources, we can give everyone a supercomputer in their pocket just to take selfies and talk about Pokemon. Why would our AIs be any different?
- xwolfi 1y agoBecause we really could use all that time, money and uranium to train them on complex problems we need to solve, rather than entertainment. But you're right, shareholders of OpenAI want profit, not progress, and they'll give us our intellectual Big Mac.
- eru 1y agoYou could say that about any piece of music or movie ever, too. Or any novel.
- deleted 1y ago[deleted]
- eru 1y agoWhy would anything change? You feed the model approximately all the text you have ever. And some things like 'popular culture of 2025' won't change, just because the calendar changed to 2026. Just like the popular culture of the 1980s is what it was, and won't change.
- int_19h 1y agoWe don't feed the model all the text ever. They are still trained on less than 1% of the entire Internet corpus.
- eru 1y agoYou are right, though on the other hand feeding it a selection of 1% of the entire corpus is already pretty close to 'all the text' (if you assume exponential growth in training over time). Even multiplying that to approximately 100% of that corpus plus adding lots of non-internet text, will pale in comparison to all the non-text training data we will (or are) feeding our coming (and existing) multi-modal models. If I may go out an a limb here: either we will see continuous great progress on text-based LLMs alone, or multi-modal models will become the next big focus. (Or both.) That's because people are hungry for progress, and going multi-modal is the obvious thing to try to focus on, if text alone proves infeasible to drive progress. Just to be clear: I make no prediction here on whether multi-modal will lead to progress, just that people will obviously try it and try it hard, if the focus on text starts to stall.
- diggan 1y ago> Why would anything change? It's not that facts change across time, but the relevancy of the details change. For example, it would be great if we could teach LLMs all the APIs all React versions have ever had, but if we do that for everything, there will be no limit to the weight's weight, and we'd need new weights every quarter if not more often. That seems very unsustainable. So the information that corresponds to "What is the current React API for X" changes whenever the API changes, but "What is the React v5 API for X" remains the same. Having the model being able to look up those things via external channels would let us use the same models for way longer, if you need "up to date data" about things.