6 ms·

’explains LeCun. “It’s not as if we’re going to be able to scale them up and train them with more data, with bigger computers, and reach human intelligence.’ W

’explains LeCun. “It’s not as if we’re going to be able to scale them up and train them with more data, with bigger computers, and reach human intelligence.’ With all due respect to LeCun, not him nor anyone else in the field predicted the new emergent capabilities brought within the last few years. So, he’s saying this is not going to keep happening? What level of confidence is he putting on not seeing it last time, but being right this time?

freeredpill 3y ago
Jurgen Schmidhuber has published on creativity as an algorithmic process for a long time. And others.
- Zetobal 3y ago
  Without anything to show but hypothesis.
croes 3y ago
Not with LLMs, because the amount of quality data is limited.
- WhitneyLand 3y ago
  Maybe, maybe not. The point is why should we take that as true when people didn’t have a clue LLMs would take us this far, until they did. Btw there are lots of ideas on how to deal with the data issue. Multiple directions in research are promising to work around that.
sinuhe69 3y ago
AFAIK, PaLM was up and running inside Google long before the hype with ChatGPT. So in the circle of frontier AI researchers, they miss know it. But Google kept silence because of the problematic hallucinations and the fear for litigation and their reputation. Besides, in his analysis, le Cun consistently spoke of GPT as “writing help, no more no, less”.
- cubefox 3y ago
  PaLM was a base model without any fine-tuning, itself a copy of GPT-3. Maybe you meant LaMDA, which was based on pure instruction fine-tuning without RLHF.
  - sinuhe69 3y ago
    Sorry but your claim is so ridiculous. PaLM employs a different model as ChatGPT. And Google also published their result in April 2022, earlier than OpenAI talked about their ChatGPT in late 2022.
    - HeatrayEnjoyer 3y ago
      I don't really care which tech corp did what first, just want to point out GPT-4 was being beta trialed to some bing users in mid 2022.
    - cubefox 3y ago
      PaLM was basically a larger version of GPT-3. It was not similar to ChatGPT. It was a pure token predictor. It was not fine-tuned for chat.
      - sinuhe69 3y ago
        PaLM use a pathway architect, not a decoder only transformer as ChatGPT. At least you can look it up on Wikipedia before making such wild assumptions.
- gmerc 3y ago
  Google didn’t saw a way to monetize the technology within their existing very profitable business model. It’s the convergence of end of ad economy/attention supply growth and Microsoft’s money forcing AI as the next growth narrative that’s forcing google to join the party - at massive cost to them.
  - solidasparagus 3y ago
nighthawk454 3y ago
What emergent capabilities exactly? I’m not sure “no one predicted this” is accurate, more like there was a lot of hype in the research community circa 2018 when this stuff developed, and it’s just not news these days research-wise
- cubefox 3y ago
  It absolutely was accurate. Back in 2018 (GPT-1) nobody expected we would soon have something as powerful as GPT-3, or even ChatGPT. The massive progress in LLMs (and text-to-image models) took everyone by surprise. They achieved things that, until recently, were regarded as pure science fiction about the distant future.
  - bamboozled 3y ago
    What’s so “powerful” about ChatGPT 3 and 4 ?
    - ruszki 3y ago
      They respond so well that a lot of people believe their results without question, even while we know that they hallucinate all the time. They’re good enough for many. Even here in HN, where everybody should know that they are terrible with numbers, majority of us cheered that somebody generated diagrams with them, I couldn’t even find a single comment which were about “wait a minute, they’re fucking unreliable numbers”. Also regarding coding, they can definitely produce junior level code, especially if you follow a similar trial and error way of thinking which happens all the time during your work, if you try to use an unknown interface.
    - lambdaone 3y ago
      Their ability to superficially mimic cognition, in a way that not only raises significant philosophical questions but also seems (rightly or wrongly) to bring the prospect of "true" AI tantalizingly closer.
      - 3y ago
jart 3y ago
> With all due respect to LeCun, not him nor anyone else in the field predicted the new emergent capabilities brought within the last few years. Turing predicted it would happen around the year 2000 and take ~1gb of ram. > I believe that in about fifty years’ time it will be possible to programme computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent, chance of making the right identification after five minutes of questioning. https://academic.oup.com/mind/article/LIX/236/433/986238 https://academic.oup.com/mind/article/LIX/236/433/986238 I think Turing was right. If I run TinyLlama 1.1B on my computer I can have a conversation where it pretends to be a person. It's small and fast enough that it'd probably run fine on a high end workstation from 2000. If the tech was possible back then, then it probably existed. Keep in mind Turing was the sort of person whose work at Bletchley Park took 30 years to declassify.
- simne 3y ago
  > Turing predicted it would happen around the year 2000 and take ~1gb of ram Turing was right, but he believed in human intelligence, so thinking linear. Unfortunately, neural networks appear harder then sought, because Forward-Forward learning with which working human brain, is now considered too computationally hard to be used practically. Reverse-Propagation neural networks are much easier computationally to achieve, but are magnitudes less dense than FF, that's why we have so much delay - just need thousand times more performance, this is about 30 additional years. PS I tried Llama 30b on my computer (honesty, tried all Lllama models could fit in 64G RAM, from 13B and more). It is not sprinter fast, but I'm very impressed, on how deep it could think. I must admit, I don't want to talk to many humans anymore, as It looks smarter.
TaylorAlexander 3y ago
Making larger and larger datasets works fine for text and images. The internet has so much of that we can scrape superhuman quantities of the stuff and shove it in to a next token predictor and the result is pretty good. But what about for tasks where datasets don’t really exist? I do a lot of PCB design and it’s extremely time consuming. But it’s a niche field compared to text and images. No dataset exists that says “these were the engineering requirements of this PCB and this is the result and by the way the board actually worked”. So how will we train AI systems to replace a human doing PCB design? It’s probably going to need to learn PCB design from first principles (along with massive help from large transformers when possible, like collections of chip datasheets). Even then, understanding PDF datasheets is something these big companies haven’t really pulled off yet, though I suspect in 5 years that will change. But my point is that there must be loads of tasks, even on the computer, for which suitable datasets don’t exist and it would be infeasible to create them. Another big thing I do is machine design and again it’s not about designing one mechanical part it’s about pulling in the right parts from all over the world and assuming certain manufacturing processes, and then knowing those processes and then designing all the parts and the assembly. There’s so many different pieces of knowledge in there that are not captured on text or images on the web and would be hard to encode in to datasets. At some point we’re going to need machine systems that learn the way people do, and that’s going to take a long time to figure out. That’s what LeCun is saying.
- PoignardAzur 3y ago
  > But what about for tasks where datasets don’t really exist? I do a lot of PCB design and it’s extremely time consuming. But it’s a niche field compared to text and images. One side project I'd love to work on one day would be gathering a dataset of Factorio maps designed to help train a floor planner. The requirements aren't the same as for making a PCB, but they're similar-ish.
  - TaylorAlexander 3y ago
    I’ve never played factorio and I’ve heard it’s complex, but one difference with video games is that they have a score of some sort (I assume factorio does). While the planner I’m sure is non trivial, the score provides some feedback that you do not get in PCB design.
kaba0 3y ago
I don’t think we have significantly more data, and useful data won’t increase just on a whim. And it’s just one of the very necessary parts.
- CamperBob2 3y ago
  And it’s just one of the very necessary parts. That remains to be seen. You wouldn't say a human lacked human-level intelligence just because they had only read every book in The Pile, rather than every book in the Library of Congress.
- theptip 3y ago
  We have loads more data available. For text, as LLMs start to converse constantly with users and receive direct feedback, this unlocks more and better data. Multimodal (image/video, audio) is barely being scratched. You likely could 100x training compute and not run out of video content. Not to mention synthetic data from simulators, compilers, and other sources of infinite ground truth data. If you listen to the big labs (OpenAI, FAIR, etc) they have started to say they are not worried about data anymore. Quite obviously, at some point you put the agents out into the real world and they can collect new training data there too; this is what all the ChatGPT plugins are allowing OpenAI to do. The plugins that give good responses which are valuable will be disproportionately popular and therefore overrepresented in the new training set of user interactions.
  - Salgat 3y ago
    Text is the most useful data we have since it is so informationally dense; it's basically audio/visual data that has already been converted by a wet neural net into useful information. Video/audio is not going to be as significant a contribution as people think when it comes to training a language model on expressing new knowledge. It's best to think of video and audio as text encoded in a very complex layer of noise.
Salgat 3y ago
I think his point might be more that we've already gave all the data and hardware that is feasibly possible with the world's richest companies; exponential scaling with that has hit its limit. Improvements (including more data and hardware) now are more incremental until the next revolutionary architecture change occurs.

freeredpill 3y ago

Jurgen Schmidhuber has published on creativity as an algorithmic process for a long time. And others.

Zetobal 3y ago
Without anything to show but hypothesis.

croes 3y ago

Not with LLMs, because the amount of quality data is limited.

WhitneyLand 3y ago
Maybe, maybe not. The point is why should we take that as true when people didn’t have a clue LLMs would take us this far, until they did. Btw there are lots of ideas on how to deal with the data issue. Multiple directions in research are promising to work around that.

sinuhe69 3y ago

AFAIK, PaLM was up and running inside Google long before the hype with ChatGPT. So in the circle of frontier AI researchers, they miss know it. But Google kept silence because of the problematic hallucinations and the fear for litigation and their reputation. Besides, in his analysis, le Cun consistently spoke of GPT as “writing help, no more no, less”.

cubefox 3y ago
PaLM was a base model without any fine-tuning, itself a copy of GPT-3. Maybe you meant LaMDA, which was based on pure instruction fine-tuning without RLHF.
- sinuhe69 3y ago
  Sorry but your claim is so ridiculous. PaLM employs a different model as ChatGPT. And Google also published their result in April 2022, earlier than OpenAI talked about their ChatGPT in late 2022.
  - HeatrayEnjoyer 3y ago
    I don't really care which tech corp did what first, just want to point out GPT-4 was being beta trialed to some bing users in mid 2022.
  - cubefox 3y ago
    PaLM was basically a larger version of GPT-3. It was not similar to ChatGPT. It was a pure token predictor. It was not fine-tuned for chat.
    - sinuhe69 3y ago
      PaLM use a pathway architect, not a decoder only transformer as ChatGPT. At least you can look it up on Wikipedia before making such wild assumptions.
gmerc 3y ago
Google didn’t saw a way to monetize the technology within their existing very profitable business model. It’s the convergence of end of ad economy/attention supply growth and Microsoft’s money forcing AI as the next growth narrative that’s forcing google to join the party - at massive cost to them.
- solidasparagus 3y ago

nighthawk454 3y ago

What emergent capabilities exactly? I’m not sure “no one predicted this” is accurate, more like there was a lot of hype in the research community circa 2018 when this stuff developed, and it’s just not news these days research-wise

cubefox 3y ago
It absolutely was accurate. Back in 2018 (GPT-1) nobody expected we would soon have something as powerful as GPT-3, or even ChatGPT. The massive progress in LLMs (and text-to-image models) took everyone by surprise. They achieved things that, until recently, were regarded as pure science fiction about the distant future.
- bamboozled 3y ago
  What’s so “powerful” about ChatGPT 3 and 4 ?
  - ruszki 3y ago
    They respond so well that a lot of people believe their results without question, even while we know that they hallucinate all the time. They’re good enough for many. Even here in HN, where everybody should know that they are terrible with numbers, majority of us cheered that somebody generated diagrams with them, I couldn’t even find a single comment which were about “wait a minute, they’re fucking unreliable numbers”. Also regarding coding, they can definitely produce junior level code, especially if you follow a similar trial and error way of thinking which happens all the time during your work, if you try to use an unknown interface.
  - lambdaone 3y ago
    Their ability to superficially mimic cognition, in a way that not only raises significant philosophical questions but also seems (rightly or wrongly) to bring the prospect of "true" AI tantalizingly closer.
    - 3y ago

jart 3y ago

> With all due respect to LeCun, not him nor anyone else in the field predicted the new emergent capabilities brought within the last few years. Turing predicted it would happen around the year 2000 and take ~1gb of ram. > I believe that in about fifty years’ time it will be possible to programme computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent, chance of making the right identification after five minutes of questioning. https://academic.oup.com/mind/article/LIX/236/433/986238 https://academic.oup.com/mind/article/LIX/236/433/986238 I think Turing was right. If I run TinyLlama 1.1B on my computer I can have a conversation where it pretends to be a person. It's small and fast enough that it'd probably run fine on a high end workstation from 2000. If the tech was possible back then, then it probably existed. Keep in mind Turing was the sort of person whose work at Bletchley Park took 30 years to declassify.

simne 3y ago
> Turing predicted it would happen around the year 2000 and take ~1gb of ram Turing was right, but he believed in human intelligence, so thinking linear. Unfortunately, neural networks appear harder then sought, because Forward-Forward learning with which working human brain, is now considered too computationally hard to be used practically. Reverse-Propagation neural networks are much easier computationally to achieve, but are magnitudes less dense than FF, that's why we have so much delay - just need thousand times more performance, this is about 30 additional years. PS I tried Llama 30b on my computer (honesty, tried all Lllama models could fit in 64G RAM, from 13B and more). It is not sprinter fast, but I'm very impressed, on how deep it could think. I must admit, I don't want to talk to many humans anymore, as It looks smarter.

TaylorAlexander 3y ago

Making larger and larger datasets works fine for text and images. The internet has so much of that we can scrape superhuman quantities of the stuff and shove it in to a next token predictor and the result is pretty good. But what about for tasks where datasets don’t really exist? I do a lot of PCB design and it’s extremely time consuming. But it’s a niche field compared to text and images. No dataset exists that says “these were the engineering requirements of this PCB and this is the result and by the way the board actually worked”. So how will we train AI systems to replace a human doing PCB design? It’s probably going to need to learn PCB design from first principles (along with massive help from large transformers when possible, like collections of chip datasheets). Even then, understanding PDF datasheets is something these big companies haven’t really pulled off yet, though I suspect in 5 years that will change. But my point is that there must be loads of tasks, even on the computer, for which suitable datasets don’t exist and it would be infeasible to create them. Another big thing I do is machine design and again it’s not about designing one mechanical part it’s about pulling in the right parts from all over the world and assuming certain manufacturing processes, and then knowing those processes and then designing all the parts and the assembly. There’s so many different pieces of knowledge in there that are not captured on text or images on the web and would be hard to encode in to datasets. At some point we’re going to need machine systems that learn the way people do, and that’s going to take a long time to figure out. That’s what LeCun is saying.

PoignardAzur 3y ago
> But what about for tasks where datasets don’t really exist? I do a lot of PCB design and it’s extremely time consuming. But it’s a niche field compared to text and images. One side project I'd love to work on one day would be gathering a dataset of Factorio maps designed to help train a floor planner. The requirements aren't the same as for making a PCB, but they're similar-ish.
- TaylorAlexander 3y ago
  I’ve never played factorio and I’ve heard it’s complex, but one difference with video games is that they have a score of some sort (I assume factorio does). While the planner I’m sure is non trivial, the score provides some feedback that you do not get in PCB design.

kaba0 3y ago

I don’t think we have significantly more data, and useful data won’t increase just on a whim. And it’s just one of the very necessary parts.

CamperBob2 3y ago
And it’s just one of the very necessary parts. That remains to be seen. You wouldn't say a human lacked human-level intelligence just because they had only read every book in The Pile, rather than every book in the Library of Congress.
theptip 3y ago
We have loads more data available. For text, as LLMs start to converse constantly with users and receive direct feedback, this unlocks more and better data. Multimodal (image/video, audio) is barely being scratched. You likely could 100x training compute and not run out of video content. Not to mention synthetic data from simulators, compilers, and other sources of infinite ground truth data. If you listen to the big labs (OpenAI, FAIR, etc) they have started to say they are not worried about data anymore. Quite obviously, at some point you put the agents out into the real world and they can collect new training data there too; this is what all the ChatGPT plugins are allowing OpenAI to do. The plugins that give good responses which are valuable will be disproportionately popular and therefore overrepresented in the new training set of user interactions.
- Salgat 3y ago
  Text is the most useful data we have since it is so informationally dense; it's basically audio/visual data that has already been converted by a wet neural net into useful information. Video/audio is not going to be as significant a contribution as people think when it comes to training a language model on expressing new knowledge. It's best to think of video and audio as text encoded in a very complex layer of noise.

Salgat 3y ago

I think his point might be more that we've already gave all the data and hardware that is feasibly possible with the world's richest companies; exponential scaling with that has hit its limit. Improvements (including more data and hardware) now are more incremental until the next revolutionary architecture change occurs.