4 ms·

I don't think anyone in research actually believes this. Note that the whole idea behind claiming "scaling laws" will infinitely improve these models is a fundi

I don't think anyone in research actually believes this. Note that the whole idea behind claiming "scaling laws" will infinitely improve these models is a funding strategy rather than a research one. None of these folks think human-like consciousness will "rise" from this effort, even though they veil it to continue the hype-cycle. I guarantee all these firms are desperately looking for architectural breakthroughs, even while they wax poetic about scaling laws, they know there is a bottleneck ahead. Notice how LeCun is the only researcher being honest about this in a public fashion. Meta is committed to AI already and will at least match the spend of competitors anyway, so he doesn't have as much pressure to try and convince investors that this rabbit whole is deeper. Don't get me wrong, LLMs are a tremendous improvement on knowledge compression and distillation, but it's still unreliable enough that old school search is likely a superior method nonetheless.

xpe 2y ago
I don’t hold LeCun’s opinions in high regard because of his often hyperbolic statements.
xpe 2y ago
Put aside consciousness or hype or investment. Look at the results; LLMs are well beyond old-school search in many ways. Sure, they are flawed in someways. Previous paradigms for search, were also flawed in their own ways. Look at the arc of NLP. Large language models fit the pattern. One could even say that their development (next token prediction with a powerful function approximator) is obvious in hindsight.
- obirunda 2y ago
  Honestly I don't disagree, I just think that humans tend to anthropomorphize to such a high extent that there is a fair bit of hyperbole promoting LLMs as more than they are. It's my opinion that the big flaws LLMs currently present aren't going to be overcome by scaling alone.
  - xpe 2y ago
    Scaling existing architectures (inference I mean) will probably help a lot. Combine that with better training and hybrid architectures, and I personally expect to see continued improvement. However, given the hype cycle, combined with broad levels of ignorance of how LLMs work, it is an open question if even amazing progress will impress people anymore.
    - obirunda 2y ago
      I'm less concerned with people's perception and strictly concerned with value. If we were to define value as the number of things that can be automated or severely improved by the technology or its future versions, there is a misalignment between value and perceived value. The value is lower than perceived because there is an assumption that what's preventing higher value delivery from the investment is for the models to get better at generating responses from prompts. But there are two issues with this position. 1. LLMs still require plenty of assistance where they are writing production ready code and making function-calls, especially if the original API wasn't designed with LLMs in mind. Unless there is a leap in architecture that's going to make all the tools we have, including non-API ones easily accessible by the models to interact with, the amount of glue code required to make it all work increases the potential for features that users can use, but not necessarily deliver higher value to developers in that a lay person still probably can't develop software with an LLM copilot in toe. So yes, no one is going to be impressed even if we see models improve further on benchmarking. 2. Long-Horizon goals. Long-term research or even project management requires interdisciplinary understanding of how all the goals around success relate to each other and most importantly how to assess if an outcome is leading to a goal accomplishment or not. There isn't an architectural foundation for the models to be grounded in a reality that presupposes these abilities. What I fail to see, is how improving next token prediction will materially move the needle on these other aspects of intelligence that aren't necessarily related to generating an output or a series of outputs orchestrated over an evolving set of requirements. Honestly lI think that the LLM portion of the human brain has likely be surpassed by existing models

xpe 2y ago

I don’t hold LeCun’s opinions in high regard because of his often hyperbolic statements.

xpe 2y ago

Put aside consciousness or hype or investment. Look at the results; LLMs are well beyond old-school search in many ways. Sure, they are flawed in someways. Previous paradigms for search, were also flawed in their own ways. Look at the arc of NLP. Large language models fit the pattern. One could even say that their development (next token prediction with a powerful function approximator) is obvious in hindsight.

obirunda 2y ago
Honestly I don't disagree, I just think that humans tend to anthropomorphize to such a high extent that there is a fair bit of hyperbole promoting LLMs as more than they are. It's my opinion that the big flaws LLMs currently present aren't going to be overcome by scaling alone.
- xpe 2y ago
  Scaling existing architectures (inference I mean) will probably help a lot. Combine that with better training and hybrid architectures, and I personally expect to see continued improvement. However, given the hype cycle, combined with broad levels of ignorance of how LLMs work, it is an open question if even amazing progress will impress people anymore.
  - obirunda 2y ago
    I'm less concerned with people's perception and strictly concerned with value. If we were to define value as the number of things that can be automated or severely improved by the technology or its future versions, there is a misalignment between value and perceived value. The value is lower than perceived because there is an assumption that what's preventing higher value delivery from the investment is for the models to get better at generating responses from prompts. But there are two issues with this position. 1. LLMs still require plenty of assistance where they are writing production ready code and making function-calls, especially if the original API wasn't designed with LLMs in mind. Unless there is a leap in architecture that's going to make all the tools we have, including non-API ones easily accessible by the models to interact with, the amount of glue code required to make it all work increases the potential for features that users can use, but not necessarily deliver higher value to developers in that a lay person still probably can't develop software with an LLM copilot in toe. So yes, no one is going to be impressed even if we see models improve further on benchmarking. 2. Long-Horizon goals. Long-term research or even project management requires interdisciplinary understanding of how all the goals around success relate to each other and most importantly how to assess if an outcome is leading to a goal accomplishment or not. There isn't an architectural foundation for the models to be grounded in a reality that presupposes these abilities. What I fail to see, is how improving next token prediction will materially move the needle on these other aspects of intelligence that aren't necessarily related to generating an output or a series of outputs orchestrated over an evolving set of requirements. Honestly lI think that the LLM portion of the human brain has likely be surpassed by existing models