4 ms·

This is very much people seeing the Apple Lisa(and/or Mac) for the first time. It had all been done better in the labs at Xerox over a decade earlier. No one go

by soperj 4y ago

This is very much people seeing the Apple Lisa(and/or Mac) for the first time. It had all been done better in the labs at Xerox over a decade earlier. No one got to use it though.

msikora 4y ago
Google is the new Xerox - tons of cool tech that nobody gets to see besides white papers and articles. And the only real product that makes money is the search engine that may get disrupted very soon by newcomers using tech based on Google's own research.
nightski 4y ago
To me it's childish, and I have no idea why Yann decided this needed to be shouted on Twitter. I've lost a lot of respect for him over this.
- arp242 4y ago
  According to the article: "LeCun made his remarks about OpenAI in response to a question during the colloquium posed by New York Times journalist Cade Metz. Metz asked if Meta's AI team, FAIR, which LeCun built, will ever be identified in the public mind with breakthroughs the way that OpenAI is." It seems to me "well actually, ChatGPT isn't really the breakthrough you think it is" is an appropriate response. He wasn't dismissive of ChatGPT and even praised it to some degree.
  - nightski 4y ago
    He's been tweeting quite a bit about it beyond what you referenced which was more what I was referring to.
- soperj 4y ago
  I think if you read the headline it seems childish. His main point is that Google/Meta couldn't really release this, since the accuracy is really really poor.
  - cl42 4y ago
    If that's true, then this is a beautiful example of creative destruction.
- visarga 4y ago
  Well, the transformer was invented at Google, language models were decades old. But scaling it was not done before, and preparing the dataset at this size, and babysitting the model so its loss doesn't explode during training - all were innovations that required lots of work and money to be achieved, so we can just copy the same formula without redoing all the steps.
jancsika 4y ago
Show me the AI chat system from a decade ago that could hallucinate a git repo. Edit: let me be more careful: Show me the AI chat system from a decade ago that wasn't specifically designed to hallucinate git repos, but would hallucinate a git repo in response a detailed natural language prompt. It should have been good enough in its output to cause the user to write more prompts in an attempt to gauge whether the AI system is querying the internet in realtime to generate its response.
- jchw 4y ago
  Decade ago, definitely not. But the answer to that question, time aside, is, well, GPT-3. No doubt ChatGPT is better, but it does seem like a decent bit of the improvement was UX related, since with the right prompts people have shown similarly miraculous output from GPT-3. Of course, I'm pretty sure the quoted person is referring to the fact that Google and others likely have similarly capable LLMs already that they simply won't release the models for. I can see the argument that new architectures and designs can be innovative, but that simply being the one to throw a million dollars of compute time into training is much less so, if that's the kind of argument.
  - hnfong 4y ago
    Yes, but along that reasoning the criticism that "ChatGPT is not particularly innovative" is a bit disingenuous in the sense that the general public has never had easy access to these models until ChatGPT arrived. Besides, even though the newer GPT models presumably just extended earlier ones with similar techniques but better hardware and data, it wasn't a foregone conclusion how powerful (from the user perspective) the models were going to be. Somebody had to put the resources into training the models. It's like saying human brains aren't particularly powerful because even the simplest animals have neurons and it's obvious that a more powerful brain could be built with more of them.
    - jchw 4y ago
      I don't think the point here is to downplay the significance of ChatGPT as a product, but to point out that OpenAI is likely not uniquely resourced in this regard. If anything, it sounds like what they're trying to say is that ChatGPT is only the tip of the iceberg, and the reason why it came out of OpenAI and not Meta or Google is more a function of other factors like reputational damage, etc. The headline does hit as a bit sensationalist or dismissive if you read it literally, as if it's claiming the work that went into ChatGPT was all easy and no big deal. Rather though, it seems like it's just trying to point out to the zeitgeist currently enamored with ChatGPT and all of the new possibilities, that other companies also have this capability today and that transformers have been more of an evolutionary process than a revolutionary process. Maybe I'm reading it too leniently, but that was my takeaway.
babypuncher 4y ago
While Xerox did it first, I think it's a stretch to say they did it better than the Macintosh.
- soperj 4y ago
  No it really isn't. https://www.quora.com/How-did-the-original-Macintosh-compare-to-the-Xerox-Alto https://www.quora.com/How-did-the-original-Macintosh-compare...
  - babypuncher 4y ago
    While the Alto was certainly more capable than the first Macintosh, I find the user experience to be less refined and more clunky. At this point though we are debating the merits of polish vs functionality and it's impossible to make an assessment either way without some degree of subjectivity.

by soperj 4y ago

This is very much people seeing the Apple Lisa(and/or Mac) for the first time. It had all been done better in the labs at Xerox over a decade earlier. No one got to use it though.

msikora 4y ago

Google is the new Xerox - tons of cool tech that nobody gets to see besides white papers and articles. And the only real product that makes money is the search engine that may get disrupted very soon by newcomers using tech based on Google's own research.

nightski 4y ago

To me it's childish, and I have no idea why Yann decided this needed to be shouted on Twitter. I've lost a lot of respect for him over this.

arp242 4y ago
According to the article: "LeCun made his remarks about OpenAI in response to a question during the colloquium posed by New York Times journalist Cade Metz. Metz asked if Meta's AI team, FAIR, which LeCun built, will ever be identified in the public mind with breakthroughs the way that OpenAI is." It seems to me "well actually, ChatGPT isn't really the breakthrough you think it is" is an appropriate response. He wasn't dismissive of ChatGPT and even praised it to some degree.
- nightski 4y ago
  He's been tweeting quite a bit about it beyond what you referenced which was more what I was referring to.
soperj 4y ago
I think if you read the headline it seems childish. His main point is that Google/Meta couldn't really release this, since the accuracy is really really poor.
- cl42 4y ago
  If that's true, then this is a beautiful example of creative destruction.
visarga 4y ago
Well, the transformer was invented at Google, language models were decades old. But scaling it was not done before, and preparing the dataset at this size, and babysitting the model so its loss doesn't explode during training - all were innovations that required lots of work and money to be achieved, so we can just copy the same formula without redoing all the steps.

jancsika 4y ago

Show me the AI chat system from a decade ago that could hallucinate a git repo. Edit: let me be more careful: Show me the AI chat system from a decade ago that wasn't specifically designed to hallucinate git repos, but would hallucinate a git repo in response a detailed natural language prompt. It should have been good enough in its output to cause the user to write more prompts in an attempt to gauge whether the AI system is querying the internet in realtime to generate its response.

jchw 4y ago
Decade ago, definitely not. But the answer to that question, time aside, is, well, GPT-3. No doubt ChatGPT is better, but it does seem like a decent bit of the improvement was UX related, since with the right prompts people have shown similarly miraculous output from GPT-3. Of course, I'm pretty sure the quoted person is referring to the fact that Google and others likely have similarly capable LLMs already that they simply won't release the models for. I can see the argument that new architectures and designs can be innovative, but that simply being the one to throw a million dollars of compute time into training is much less so, if that's the kind of argument.
- hnfong 4y ago
  Yes, but along that reasoning the criticism that "ChatGPT is not particularly innovative" is a bit disingenuous in the sense that the general public has never had easy access to these models until ChatGPT arrived. Besides, even though the newer GPT models presumably just extended earlier ones with similar techniques but better hardware and data, it wasn't a foregone conclusion how powerful (from the user perspective) the models were going to be. Somebody had to put the resources into training the models. It's like saying human brains aren't particularly powerful because even the simplest animals have neurons and it's obvious that a more powerful brain could be built with more of them.
  - jchw 4y ago
    I don't think the point here is to downplay the significance of ChatGPT as a product, but to point out that OpenAI is likely not uniquely resourced in this regard. If anything, it sounds like what they're trying to say is that ChatGPT is only the tip of the iceberg, and the reason why it came out of OpenAI and not Meta or Google is more a function of other factors like reputational damage, etc. The headline does hit as a bit sensationalist or dismissive if you read it literally, as if it's claiming the work that went into ChatGPT was all easy and no big deal. Rather though, it seems like it's just trying to point out to the zeitgeist currently enamored with ChatGPT and all of the new possibilities, that other companies also have this capability today and that transformers have been more of an evolutionary process than a revolutionary process. Maybe I'm reading it too leniently, but that was my takeaway.

babypuncher 4y ago

While Xerox did it first, I think it's a stretch to say they did it better than the Macintosh.

soperj 4y ago
No it really isn't. https://www.quora.com/How-did-the-original-Macintosh-compare-to-the-Xerox-Alto https://www.quora.com/How-did-the-original-Macintosh-compare...
- babypuncher 4y ago
  While the Alto was certainly more capable than the first Macintosh, I find the user experience to be less refined and more clunky. At this point though we are debating the merits of polish vs functionality and it's impossible to make an assessment either way without some degree of subjectivity.