5 ms·
To me, academia has always been about developing proof-of-concepts for techniques that industry adopts years later. These proof-of-concepts are intricate but sm
by armchairhacker 3y ago
To me, academia has always been about developing proof-of-concepts for techniques that industry adopts years later. These proof-of-concepts are intricate but small, so they require a lot of ingenuity but not much resources or man-hours (a company can also hire far more employees than a professor).
AI is no exception. All evidence suggests that current LLMs don't scale (EDIT: they do scale, but at a certain point somewhere around GPT4 the scaling slows down very quickly), so we need new techniques to fix their (many) flaws. A proof-of-concept for a new technique doesn't need hundreds of millions of dollars to demonstrate huge potential: such a demonstration only needs a small model using the new technique, a small model using the current SOTA, and some evidence that it scales (like slightly larger models that show the new model isn't slowing down its scaling vs SOTA).
Academia is also about creating better explanations for things we already have. So researchers don't even need to develop new models, they can simply create small existing models and demonstrate some new property or explanation to get a worthwhile result. We probably need better explanations for how and why current LLMs work in order to create the better models mentioned above.
EDIT: At least how it's supposed to work, you don't even need to show success either. Academics merely need to show a new technique with the potential to improve the SOTA. Even if the technique is a huge failure, the work still advances the future of AI, by contributing to our understanding of models (what works and what doesn't) as well as removing one potential option.
- famouswaffles 3y ago>AI is no exception. All evidence suggests that current LLMs don't scale Lol that's the opposite of what the evidence shows.
- BenFranklin100 3y agoYou got it exactly right. Academics have the freedom to pursue avenues and interests that have no clear immediate application nor profit.
- pama 3y agoI have seen no evidence that LLMs don’t scale. What evidence did you have in mind? Do you mean something specific and perhaps different from what I would call scaling? It seems in the last two years people got better at squeezing out more performance from smaller (but still large) language models but the continuous scaling of large models to multi modal applications has been rather impressive.
- wavemode 3y agoThey didn't say LLM's don't scale. They said current LLM's don't scale, which appears to be true. LLM reasoning ability appears to be plateauing pretty hard, no matter how much we spend.
- pama 3y agoScaling in this broader context can have many meanings. All of them disagree with the original statement and with yours. If you assume that OpenAI spent more on GPT3.5 than on GPT3, and more on GPT4 than on GPT3.5, which is certainly plausible, how do I understand your statement in that view? What is even meant with scaling stopping at GPT4 when the next model by OpenAI is not yet public? Of course it makes no sense to compare against yet another company throwing money at a problem, but within each company scaling to more money (training/modalities/tunings) as a function of time has improved reasoning abilities. I am probably misunderstanding something simple so any citation would help.
- deadbabe 3y agoI’m glad things are plateauing because when I suggested this could happen I was written off as an idiot. People just kept pointing to the leap from GPT 2 to GPT 3 and GPT 3 to GPT 4 and extrapolating that clearly progress will be exponential forever, some of those people were on this very site.
- famouswaffles 3y agoThere's pretty much nothing to suggest things are plateauing lol
- xadhominemx 3y agoThat GPT, Gemini, and Claude are all about the level of GPT-4 is evidence that things are plateauing -- not conclusive evidence, but evidence nonetheless.
- 3y ago
- humansareok1 3y ago>All evidence suggests that current LLMs don't scale (EDIT: they do scale, but at a certain point somewhere around GPT4 the scaling slows down very quickly) There seems to actually be no evidence that LLMs dont scale. All data from LLMs leading up to GPT4 indicates they scale incredibly well actually. Do you have a single paper you can point to that says otherwise?
- 4death4 3y agoNo OP, but no amount of data will ever enable an LLM to know when it doesn’t know something, for instance. That’s a fundamental limitation with the current architecture.
- bilbo0s 3y agoBut that's different from scaling. You can not know what you don't know, and learn gobs of new stuff at the same time. That's the natural state of most learners.
- deleted 3y ago[deleted]
- datameta 3y agoWhen we have an external source of information that is not self-accountable, it is different from when we are both the learner and functor - when we notice mistakes we learn from them. We're always looking out for errors, and if they happen then (usually) we alone feel responsible for their effects.
- bilbo0s 3y agoAgain, you're talking about something different than scaling. There are many different aspects of emotions, life, or even learning. But the comment was about scaling learning, which can be unaffected by other aspects, such as whether you learned something incorrect. More succinctly: I can still teach Simpson's Rule in a Calculus class to a young man or woman who subscribes to Creationism, or the Flat Earth theory. None of their false knowledge affects their innate ability to take in new knowledge.
- blackbear_ 3y ago> Academics merely need to show a new technique with the potential to improve the SOTA. Not sure which field you are talking about, but I've had more than one machine learning paper rejected because it didn't improve SOTA enough. Not to contradict your point directly, but current publication practices don't seem to scale very well with the progress in AI
- lupire 3y agoEvery paper get rejected somewhere. The prestige journal industry is growing more slowly than the AI industry.