8 ms·

Yann LeCun: ChatGPT is 'not particularly innovative'

78 points by jonbaer 4y ago

Reimersholme 4y ago
[dead]
alephxyz 4y ago
He's pulling a Schmidhuber.
zackees 4y ago
[dead]
deleted 4y ago
[deleted]
timcavel 4y ago
[dead]
visarga 4y ago
Technically, he is right. But OpenAI was the first to find the winning formula and make it famous.
- deleted 4y ago
  [deleted]
- KrazyButTrue 4y ago
  What do you mean by "winning formula" in this sentence?
  - Zanneth 4y ago
    Making a product and shipping it.
  - consumer451 4y ago
    Not the parent, but to me it generally means 1 million (likely non-dev, regular people) users in 5 days. For me personally: I have had success asking for code snippets and general brainstorming in areas which are not my forte, all the while using a very nice and clean UI.
soperj 4y ago
This is very much people seeing the Apple Lisa(and/or Mac) for the first time. It had all been done better in the labs at Xerox over a decade earlier. No one got to use it though.
- msikora 4y ago
  Google is the new Xerox - tons of cool tech that nobody gets to see besides white papers and articles. And the only real product that makes money is the search engine that may get disrupted very soon by newcomers using tech based on Google's own research.
- nightski 4y ago
  To me it's childish, and I have no idea why Yann decided this needed to be shouted on Twitter. I've lost a lot of respect for him over this.
  - arp242 4y ago
    According to the article: "LeCun made his remarks about OpenAI in response to a question during the colloquium posed by New York Times journalist Cade Metz. Metz asked if Meta's AI team, FAIR, which LeCun built, will ever be identified in the public mind with breakthroughs the way that OpenAI is." It seems to me "well actually, ChatGPT isn't really the breakthrough you think it is" is an appropriate response. He wasn't dismissive of ChatGPT and even praised it to some degree.
    - nightski 4y ago
      He's been tweeting quite a bit about it beyond what you referenced which was more what I was referring to.
  - soperj 4y ago
    I think if you read the headline it seems childish. His main point is that Google/Meta couldn't really release this, since the accuracy is really really poor.
stackedinserter 4y ago
"Dropbox is just rsync + svn"
- deleted 4y ago
  [deleted]
- bryanlarsen 4y ago
  "No wireless. Less space than a nomad. Lame."
rockemsockem 4y ago
A tweet isn't worth a news article. Geeze.
- browningstreet 4y ago
  That seems to be a lot of news articles these days. I'm not particularly excited by ChatGPT, but the mercenary in me is impressed at the investment from MS it generated. From a business perspective, it accomplished "and profit!". Whether it pays dividends or reaches a richer/more-sophisticated milestone is to be seen. Personally, I think it'll muddy and flood the "bullshit" & truthiness layer pervading news, social media, auto-generated Youtube content and blogs, our digitally mediated life, etc. It's essentially the next level "bot", and those aren't held in high regard, as effective as they are. Ultimately, they're noise. I'll go a step further: ChatGPT feels more like VR & crypto than it does.. say, cloud.
  - visarga 4y ago
    It is a tool, like a knife. You can use it to reword your rough notes into an article, or to generate spam and fake news. It is a great idea fountain, but you need to filter everything with your own head to make anything useful of it.
    - browningstreet 4y ago
      I can do that for myself, but the power of tools like this.. when they bend every cultural outlet around you, changes the world. To the original point I was responding: a lot of reporting and journalism is now adding 1000 words of context to 280 word tweets. Reality context, bent.
mattnewton 4y ago
Sure, the ideas could be old (in ml research timeframes) but it works and they shipped it. It’s most people’s first exposure to a lot of innovative ideas that make up LLM at scale. I agree with his assertion that the only reason other people haven’t seen this out of Meta or Google is that they “have a lot to lose” from putting out bots “that make stuff up” - that’s the root of Google seemingly losing the narrative war I think we’re seeing here. Not sure how to get them to work through that fear.
- password11 4y ago
  >Google seemingly losing the narrative war I think we’re seeing here. Google doesn't care about the narrative war. They care about getting sued by the EU. They can put out a similar model whenever they want. The just don't want to, because the thing about AI research (vs., say, social media) is there's no first mover advantage. The best model wins and there's no network effect to bolster incumbents.
  - mattnewton 4y ago
    > The just don't want to, because the thing about AI research (vs., say, social media) is there's no first mover advantage. That’s true and I am not worried for Google in the slightest. However, once a tool is good enough, both the branding and use of it becomes somewhat sticky. It’s why people still call any tablet an iPad, and it’s one of the reasons even if a large search engine launched tomorrow people would keep using google for a long time. The problem isn’t being first, it’s if they crack changing consumer habits while google is biding their time.
    - deleted 4y ago
      [deleted]
- visarga 4y ago
  > they “have a lot to lose” from putting out bots “that make stuff up” 2023 should be the year of AI validation. For starters, we could extract facts from all the training examples (trillions) and organise them in a knowledge base. Where there is inconsistency or variation, the model should learn the distribution, so later it can confidently say a fact is not in its training data or is controversial. Note that I didn't say the model should find the truth, just the distribution of "facts". We can add source reputation signals to this KB to improve its alignment with truth. And if it is controversies we want to know, we can look at news headlines, they usually reference facts being debated. So we can know a fact is contested and by who. Another interesting approach - probing the model for "truth" by identifying a direction in the latent space that aligns with it. The logic being, even when the model is deceptive, it has an interest to know it is not telling the truth to keep the narrative consistent. So we need to just identify this signal. This is also part of work for AI alignment. I hope this year we will see large investments in validation, because the unverified outputs of a generative model are worthless and they know it. At the very least hook the model up with a calculator and a way to query the web.
cl42 4y ago
It's amazing how many researchers underestimate the importance of UX and design. Personally, seeing non-technical people using prompts for the first time -- and getting results that make sense -- is so incredible. Their eyes light up, they are surprised, and they want to keep playing with it. Amazing! A sizeable part of the population just got access to an incredibly complex AI model and can play with it. What a collective experience.
- Turing_Machine 4y ago
  Indeed. This reminds me of the people who were unimpressed by the iPod because there were other MP3 players already on the market. Sometimes "making the first X that doesn't suck" is a lot more important than "making the first X".
- ryanSrich 4y ago
  > It's amazing how many researchers underestimate the importance of UX and design. Yes. Also the fact that ChatGPT's UX and design leaves much to be desired. They could add/improve the product in so many obvious ways. I wonder if they either 1.) don't care, or 2.) have plans to, but are holding off until they take on a subscription fee.
  - Turing_Machine 4y ago
    Isn't there an API?
  - cl42 4y ago
    What would you change to the UX and design to make it even better?
    - baxtr 4y ago
      The slow spell out is really annoying. If they want to rate limit fine. But give me the answer faster and then lock queries for a few seconds instead.
      - 4y ago
jti107 4y ago
well he is right that the techniques used have been around. but this is the first mainstream AI that most people outside of the AI community have been impressed with and found useful to do real world stuff
time_to_smile 4y ago
He's not wrong. The key ingredient in ChatGPT is not brilliance but capital. It takes a lot of money not just to do the raw training but to get all the data together and process everything. There's no break through insights that make ChatGPT work, just a lot of consolidated wealth. The hacker part of me finds AI less and less interesting for this reason. We're seeing what the limits of pouring resources into the problem are. The results are cool, but I think we'll very soon see progress bound since we're using most of the data we can and it doesn't look like adding even more parameters to these models might not yield that much of result. But this shift in AI means it's increasing something programmers consume rather than produce. As far as interesting AI goes, I found the series of posts on stuff people were doing with Prolog that showed up during the last month much more interesting.
- woah 4y ago
  There are a lot of things that require an amount of processing power that the average person doesn't yet possess. This doesn't mean that they aren't interesting. 50 years ago you would have said "these computer things aren't interesting, only big institutions and corporations have them".
- sharemywin 4y ago
  I find it the opposite. There were a lot of projects I didn't have the time to work on and got bored to fast learning. So, now I can tell chatgpt to build it for me. and I only have to debug the 1 or 2 things it can't do. I built a blogging platform, a voice based chatbot using gpt3, something that takes blog article and trims out all the garbage html so only the basic html is left. created a python api, and called it from shared hosting using php. something that trims the times out of youtube captions so I can post it into chatgpt to give me an outline. I just tell it what I want the function to do and it does it. It's helped me way up my own personal productivity and creativity.
- visarga 4y ago
  When language models run out of more trillions of words to train, there is one way ahead - we need to generate more. But wait, you might say, garbage in garbage out. It won't work. Normally it wouldn't, but we add an extra ingredient here. We get a validation signal. This is problem specific, but for code it would mean to integrate the LM with a compiler and runtime so it can iterate until it solves the task, step by step. For other tasks it could mean hooking the AI to simulators, games and robots to solve tasks. It is also possible to use LLMs as simulators of text. Basically doing Reinforcement Learning with a Language Model and not just for human preferences, but for problem solving on a grand scale. Collect data from problem solving, train on it, and iterate. It costs just electricity, but LLMs can make their own data. Anthropic's Constitutional AI which is RLAIF - reinforcement learning from AI feedback is proof it can be done.
karmasimida 4y ago
He will change his tune if ChatGPT uses CNN
lofatdairy 4y ago
I mean, ChatGPT isn't even innovative for OpenAI's team. In terms of raw language capabilities it's not much more effective as GPT-3, which passed the HN Turing test almost a year before. Some of the credit goes to the fact that Dall-E 2 got everyone looking at OpenAI (and wrt Dall-E I'm personally not aware of any comparable work done anywhere else), a lot of the credit goes to the fact that OpenAI was willing to put it out there, and the work they did to make ChatGPT much safer really doesn't get acknowledged enough (on release, ChatGPT would refuse to produce potentially hateful/harmful text, and eventually a lot of the workarounds got patched as well). It's probably not fair to characterize the discourse around ChatGPT as treating it as a revolutionary bit of tech, lay people are taking note because 1) people they know are actually using it and 2) the interface really is intuitive - literally anyone knows how to use a chatbot.
- sharemywin 4y ago
  Putting it behind a login and captcha was the brilliant part.
- lostmsu 4y ago
  > In terms of raw language capabilities it's not much more effective as GPT-3, which passed the HN Turing test almost a year before. No, its ability to follow instructions is much better than GPT-3.
robga 4y ago
At least it shows Yann LeCun and Gary Marcus can agree on some things. Thankfully someone has gone full circle and had ChatGPT construct a rap battle between those minds: https://mobile.twitter.com/k_saifullaah/status/1602073411077169152 https://mobile.twitter.com/k_saifullaah/status/1602073411077...
uptownfunk 4y ago
This just reads as someone being bitter for not having developed it themselves.
cma 4y ago
It is notable that LeCun didn't credit Schmidhuber's labs' fast weight programmers research here and mostly focused on individuals and labs that got the Turing award alongside him.
- civilized 4y ago
  > Schmidhuber's labs' fast weight programmers research What does this refer to?
password11 4y ago
Full quote: "In terms of underlying techniques, ChatGPT is not particularly innovative," said Yann LeCun, Meta's chief AI scientist, in a small gathering of press and executives on Zoom last week. "It's nothing revolutionary, although that's the way it's perceived in the public," said LeCun. "It's just that, you know, it's well put together, it's nicely done."
- november84 4y ago
  So one is a bit salty no? It's not free from error but what it's able to put together is pretty great...
  - deleted 4y ago
    [deleted]
remote_phone 4y ago
Lots of salty researchers are shitting on ChatGPT meandering every single person I know is shocked. If this isn’t so innovative why didn’t they release something sooner?
- KrazyButTrue 4y ago
  If you actually read the article he answers your question directly. If you are too lazy to scroll to the bottom here's a link to a tweet: https://twitter.com/ylecun/status/1617908306420600833?s=20&t=UobdJ59kRJRhqdoDvdWwmw https://twitter.com/ylecun/status/1617908306420600833?s=20&t...
  - yesenadam 4y ago
    Be kind. Don't be snarky. ...Edit out swipes. ...Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
drcode 4y ago
Yann's tweets on this are pretty funny. I would flippantly summarize them as: "Don't get me wrong, the openai people are great. All I'm doing is educating the media, who have the mistaken belief that chatGPT is good" https://twitter.com/ylecun/status/1617921903934726144?t=2wZZ80M4n--1lPisGvNV0g&s=19 https://twitter.com/ylecun/status/1617921903934726144?t=2wZZ...
- lostmsu 4y ago
  Please don't put in quotes something that is not a direct quote.
  - cryptonector 4y ago
    > > I would flippantly summarize them as: "..." > Please don't put in quotes something that is not a direct quote. What part of "I would flippantly summarize them as" was unclear? Or was that comment edited?
    - lostmsu 4y ago
      Looks like I missed that part. But I would not call the summarization faithful to the source. The garbage bit is basically made up.
      - cryptonector 4y ago
        It's called opinion.
swatcoder 4y ago
I think what’s really being emphasized here is that ChatGPT is an first generation product without much of a novel technology moat compared to what others with capital can deliver. Microsoft will soak this work up as a pillar of their own AI assistance tech, but the other big tech firms (and probably some startups) are positioned to try to leapfrog what we see here, now that the market has been demonstrated. In the language of tech history, ChatGPT is the Lycos or Altavista to some not-yet-launched Google.
mshake2 4y ago
Sounds like somebody's job depends on convincing people there's a good reason he didn't build it first.
rcme 4y ago
AI researchers tend to have blinders when new techniques are released that don't fit their previous experience. For instance, perhaps the specific architecture of ChatGPT isn't that innovative (although I think it is), but training such a large model on so much data is innovative. Clearly this is going to become the norm going forward. Just as a personal anecdote. I worked on a ML research team at a FAANG in the early 2010s, just as NNs were becoming popular. Everyone on the ML research team had come to prominence before NNs, so they all specialized in different areas of ML. Nearly every researchers there treated NNs like some fad that would pass and thought they wouldn't be much better than the current techniques. How wrong they were... Of course, they're all working on NNs now.
drewcoo 4y ago
LeCun confuses the idea invention with that of innovation. Revolutions don't happen because suddenly there's a new idea, but because of one that catches on. https://www.wired.com/insights/2015/01/innovation-vs-invention/ https://www.wired.com/insights/2015/01/innovation-vs-inventi... I think he also confuses his jealousy with insight.
deleted 4y ago
[deleted]
tonymet 4y ago
In it's current iteration ChatGPT is more overhyped than crypto , and that says a lot
__seka__ 4y ago
Sometimes the innovation lies within the simple fact of how well known things are combined in order to create (good) products. "Not particularly innovative" is not really of concern. Besides OpenAI most likely received the media attention they wanted. Looking forward to the products Meta is going to release soon.
dayeye2006 4y ago
He thinks the eng work behind that makes the model training and inference possible is not so impressive as well...
andrewdubinsky 4y ago
They talked about ChatGPT by name in almost every one of my kids classes. Deep whatever may be further ahead, but no one knows the name of that thing. Everyone else is playing catch up now.
seydor 4y ago
It seems he s trying to say something that is technically correct, but exactly at the wrong time. It s going to be a lot of backlash if you go against the hype wave that we are riding. Better to explain in a more nuanced way how chatGPT managed to become so popular despite being an incremental improvement of fundamental findings of the past few years. People love it because it understands language, every time and without fail. So what if it spits out lies, this thing makes people happy, which is the value at play here
boto3 4y ago
Zuck probably just massaged him: Yann we should be cool as ChatGPT.
ACV001 4y ago
I think he just switched team and now he's on the Facebook twitter marketing team.
SillyUsername 4y ago
Sounds like bitter lemons to me from a team that's getting grief from Zuck for being shit and not getting there first or having the sense to. "First they ignore you, then they laugh at you, then they fight you, then you win"
xnx 4y ago
Useful >> Innovative
georgeburdell 4y ago
Ivory tower type calls good engineering ‘not particularly innovative’
- deleted 4y ago
  [deleted]

78 points by jonbaer 4y ago

Reimersholme 4y ago

[dead]

alephxyz 4y ago

He's pulling a Schmidhuber.

zackees 4y ago

[dead]

deleted 4y ago

[deleted]

timcavel 4y ago

[dead]

visarga 4y ago

Technically, he is right. But OpenAI was the first to find the winning formula and make it famous.

deleted 4y ago
[deleted]
KrazyButTrue 4y ago
What do you mean by "winning formula" in this sentence?
- Zanneth 4y ago
  Making a product and shipping it.
- consumer451 4y ago
  Not the parent, but to me it generally means 1 million (likely non-dev, regular people) users in 5 days. For me personally: I have had success asking for code snippets and general brainstorming in areas which are not my forte, all the while using a very nice and clean UI.

soperj 4y ago

This is very much people seeing the Apple Lisa(and/or Mac) for the first time. It had all been done better in the labs at Xerox over a decade earlier. No one got to use it though.

msikora 4y ago
Google is the new Xerox - tons of cool tech that nobody gets to see besides white papers and articles. And the only real product that makes money is the search engine that may get disrupted very soon by newcomers using tech based on Google's own research.
nightski 4y ago
To me it's childish, and I have no idea why Yann decided this needed to be shouted on Twitter. I've lost a lot of respect for him over this.
- arp242 4y ago
  According to the article: "LeCun made his remarks about OpenAI in response to a question during the colloquium posed by New York Times journalist Cade Metz. Metz asked if Meta's AI team, FAIR, which LeCun built, will ever be identified in the public mind with breakthroughs the way that OpenAI is." It seems to me "well actually, ChatGPT isn't really the breakthrough you think it is" is an appropriate response. He wasn't dismissive of ChatGPT and even praised it to some degree.
  - nightski 4y ago
    He's been tweeting quite a bit about it beyond what you referenced which was more what I was referring to.
- soperj 4y ago
  I think if you read the headline it seems childish. His main point is that Google/Meta couldn't really release this, since the accuracy is really really poor.

stackedinserter 4y ago

"Dropbox is just rsync + svn"

deleted 4y ago
[deleted]
bryanlarsen 4y ago
"No wireless. Less space than a nomad. Lame."

rockemsockem 4y ago

A tweet isn't worth a news article. Geeze.

browningstreet 4y ago
That seems to be a lot of news articles these days. I'm not particularly excited by ChatGPT, but the mercenary in me is impressed at the investment from MS it generated. From a business perspective, it accomplished "and profit!". Whether it pays dividends or reaches a richer/more-sophisticated milestone is to be seen. Personally, I think it'll muddy and flood the "bullshit" & truthiness layer pervading news, social media, auto-generated Youtube content and blogs, our digitally mediated life, etc. It's essentially the next level "bot", and those aren't held in high regard, as effective as they are. Ultimately, they're noise. I'll go a step further: ChatGPT feels more like VR & crypto than it does.. say, cloud.
- visarga 4y ago
  It is a tool, like a knife. You can use it to reword your rough notes into an article, or to generate spam and fake news. It is a great idea fountain, but you need to filter everything with your own head to make anything useful of it.
  - browningstreet 4y ago
    I can do that for myself, but the power of tools like this.. when they bend every cultural outlet around you, changes the world. To the original point I was responding: a lot of reporting and journalism is now adding 1000 words of context to 280 word tweets. Reality context, bent.

mattnewton 4y ago

Sure, the ideas could be old (in ml research timeframes) but it works and they shipped it. It’s most people’s first exposure to a lot of innovative ideas that make up LLM at scale. I agree with his assertion that the only reason other people haven’t seen this out of Meta or Google is that they “have a lot to lose” from putting out bots “that make stuff up” - that’s the root of Google seemingly losing the narrative war I think we’re seeing here. Not sure how to get them to work through that fear.

password11 4y ago
>Google seemingly losing the narrative war I think we’re seeing here. Google doesn't care about the narrative war. They care about getting sued by the EU. They can put out a similar model whenever they want. The just don't want to, because the thing about AI research (vs., say, social media) is there's no first mover advantage. The best model wins and there's no network effect to bolster incumbents.
- mattnewton 4y ago
  > The just don't want to, because the thing about AI research (vs., say, social media) is there's no first mover advantage. That’s true and I am not worried for Google in the slightest. However, once a tool is good enough, both the branding and use of it becomes somewhat sticky. It’s why people still call any tablet an iPad, and it’s one of the reasons even if a large search engine launched tomorrow people would keep using google for a long time. The problem isn’t being first, it’s if they crack changing consumer habits while google is biding their time.
  - deleted 4y ago
    [deleted]
visarga 4y ago
> they “have a lot to lose” from putting out bots “that make stuff up” 2023 should be the year of AI validation. For starters, we could extract facts from all the training examples (trillions) and organise them in a knowledge base. Where there is inconsistency or variation, the model should learn the distribution, so later it can confidently say a fact is not in its training data or is controversial. Note that I didn't say the model should find the truth, just the distribution of "facts". We can add source reputation signals to this KB to improve its alignment with truth. And if it is controversies we want to know, we can look at news headlines, they usually reference facts being debated. So we can know a fact is contested and by who. Another interesting approach - probing the model for "truth" by identifying a direction in the latent space that aligns with it. The logic being, even when the model is deceptive, it has an interest to know it is not telling the truth to keep the narrative consistent. So we need to just identify this signal. This is also part of work for AI alignment. I hope this year we will see large investments in validation, because the unverified outputs of a generative model are worthless and they know it. At the very least hook the model up with a calculator and a way to query the web.

cl42 4y ago

It's amazing how many researchers underestimate the importance of UX and design. Personally, seeing non-technical people using prompts for the first time -- and getting results that make sense -- is so incredible. Their eyes light up, they are surprised, and they want to keep playing with it. Amazing! A sizeable part of the population just got access to an incredibly complex AI model and can play with it. What a collective experience.

Turing_Machine 4y ago
Indeed. This reminds me of the people who were unimpressed by the iPod because there were other MP3 players already on the market. Sometimes "making the first X that doesn't suck" is a lot more important than "making the first X".
ryanSrich 4y ago
> It's amazing how many researchers underestimate the importance of UX and design. Yes. Also the fact that ChatGPT's UX and design leaves much to be desired. They could add/improve the product in so many obvious ways. I wonder if they either 1.) don't care, or 2.) have plans to, but are holding off until they take on a subscription fee.
- Turing_Machine 4y ago
  Isn't there an API?
- cl42 4y ago
  What would you change to the UX and design to make it even better?
  - baxtr 4y ago
    The slow spell out is really annoying. If they want to rate limit fine. But give me the answer faster and then lock queries for a few seconds instead.
    - 4y ago

jti107 4y ago

well he is right that the techniques used have been around. but this is the first mainstream AI that most people outside of the AI community have been impressed with and found useful to do real world stuff

time_to_smile 4y ago

He's not wrong. The key ingredient in ChatGPT is not brilliance but capital. It takes a lot of money not just to do the raw training but to get all the data together and process everything. There's no break through insights that make ChatGPT work, just a lot of consolidated wealth. The hacker part of me finds AI less and less interesting for this reason. We're seeing what the limits of pouring resources into the problem are. The results are cool, but I think we'll very soon see progress bound since we're using most of the data we can and it doesn't look like adding even more parameters to these models might not yield that much of result. But this shift in AI means it's increasing something programmers consume rather than produce. As far as interesting AI goes, I found the series of posts on stuff people were doing with Prolog that showed up during the last month much more interesting.

woah 4y ago
There are a lot of things that require an amount of processing power that the average person doesn't yet possess. This doesn't mean that they aren't interesting. 50 years ago you would have said "these computer things aren't interesting, only big institutions and corporations have them".
sharemywin 4y ago
I find it the opposite. There were a lot of projects I didn't have the time to work on and got bored to fast learning. So, now I can tell chatgpt to build it for me. and I only have to debug the 1 or 2 things it can't do. I built a blogging platform, a voice based chatbot using gpt3, something that takes blog article and trims out all the garbage html so only the basic html is left. created a python api, and called it from shared hosting using php. something that trims the times out of youtube captions so I can post it into chatgpt to give me an outline. I just tell it what I want the function to do and it does it. It's helped me way up my own personal productivity and creativity.
visarga 4y ago
When language models run out of more trillions of words to train, there is one way ahead - we need to generate more. But wait, you might say, garbage in garbage out. It won't work. Normally it wouldn't, but we add an extra ingredient here. We get a validation signal. This is problem specific, but for code it would mean to integrate the LM with a compiler and runtime so it can iterate until it solves the task, step by step. For other tasks it could mean hooking the AI to simulators, games and robots to solve tasks. It is also possible to use LLMs as simulators of text. Basically doing Reinforcement Learning with a Language Model and not just for human preferences, but for problem solving on a grand scale. Collect data from problem solving, train on it, and iterate. It costs just electricity, but LLMs can make their own data. Anthropic's Constitutional AI which is RLAIF - reinforcement learning from AI feedback is proof it can be done.

karmasimida 4y ago

He will change his tune if ChatGPT uses CNN

lofatdairy 4y ago

I mean, ChatGPT isn't even innovative for OpenAI's team. In terms of raw language capabilities it's not much more effective as GPT-3, which passed the HN Turing test almost a year before. Some of the credit goes to the fact that Dall-E 2 got everyone looking at OpenAI (and wrt Dall-E I'm personally not aware of any comparable work done anywhere else), a lot of the credit goes to the fact that OpenAI was willing to put it out there, and the work they did to make ChatGPT much safer really doesn't get acknowledged enough (on release, ChatGPT would refuse to produce potentially hateful/harmful text, and eventually a lot of the workarounds got patched as well). It's probably not fair to characterize the discourse around ChatGPT as treating it as a revolutionary bit of tech, lay people are taking note because 1) people they know are actually using it and 2) the interface really is intuitive - literally anyone knows how to use a chatbot.

sharemywin 4y ago
Putting it behind a login and captcha was the brilliant part.
lostmsu 4y ago
> In terms of raw language capabilities it's not much more effective as GPT-3, which passed the HN Turing test almost a year before. No, its ability to follow instructions is much better than GPT-3.

robga 4y ago

At least it shows Yann LeCun and Gary Marcus can agree on some things. Thankfully someone has gone full circle and had ChatGPT construct a rap battle between those minds: https://mobile.twitter.com/k_saifullaah/status/1602073411077169152 https://mobile.twitter.com/k_saifullaah/status/1602073411077...

uptownfunk 4y ago

This just reads as someone being bitter for not having developed it themselves.

cma 4y ago

It is notable that LeCun didn't credit Schmidhuber's labs' fast weight programmers research here and mostly focused on individuals and labs that got the Turing award alongside him.

civilized 4y ago
> Schmidhuber's labs' fast weight programmers research What does this refer to?

password11 4y ago

Full quote: "In terms of underlying techniques, ChatGPT is not particularly innovative," said Yann LeCun, Meta's chief AI scientist, in a small gathering of press and executives on Zoom last week. "It's nothing revolutionary, although that's the way it's perceived in the public," said LeCun. "It's just that, you know, it's well put together, it's nicely done."

november84 4y ago
So one is a bit salty no? It's not free from error but what it's able to put together is pretty great...
- deleted 4y ago
  [deleted]

remote_phone 4y ago

Lots of salty researchers are shitting on ChatGPT meandering every single person I know is shocked. If this isn’t so innovative why didn’t they release something sooner?

KrazyButTrue 4y ago
If you actually read the article he answers your question directly. If you are too lazy to scroll to the bottom here's a link to a tweet: https://twitter.com/ylecun/status/1617908306420600833?s=20&t=UobdJ59kRJRhqdoDvdWwmw https://twitter.com/ylecun/status/1617908306420600833?s=20&t...
- yesenadam 4y ago
  Be kind. Don't be snarky. ...Edit out swipes. ...Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

drcode 4y ago

Yann's tweets on this are pretty funny. I would flippantly summarize them as: "Don't get me wrong, the openai people are great. All I'm doing is educating the media, who have the mistaken belief that chatGPT is good" https://twitter.com/ylecun/status/1617921903934726144?t=2wZZ80M4n--1lPisGvNV0g&s=19 https://twitter.com/ylecun/status/1617921903934726144?t=2wZZ...

lostmsu 4y ago
Please don't put in quotes something that is not a direct quote.
- cryptonector 4y ago
  > > I would flippantly summarize them as: "..." > Please don't put in quotes something that is not a direct quote. What part of "I would flippantly summarize them as" was unclear? Or was that comment edited?
  - lostmsu 4y ago
    Looks like I missed that part. But I would not call the summarization faithful to the source. The garbage bit is basically made up.
    - cryptonector 4y ago
      It's called opinion.

swatcoder 4y ago

I think what’s really being emphasized here is that ChatGPT is an first generation product without much of a novel technology moat compared to what others with capital can deliver. Microsoft will soak this work up as a pillar of their own AI assistance tech, but the other big tech firms (and probably some startups) are positioned to try to leapfrog what we see here, now that the market has been demonstrated. In the language of tech history, ChatGPT is the Lycos or Altavista to some not-yet-launched Google.

mshake2 4y ago

Sounds like somebody's job depends on convincing people there's a good reason he didn't build it first.

rcme 4y ago

AI researchers tend to have blinders when new techniques are released that don't fit their previous experience. For instance, perhaps the specific architecture of ChatGPT isn't that innovative (although I think it is), but training such a large model on so much data is innovative. Clearly this is going to become the norm going forward. Just as a personal anecdote. I worked on a ML research team at a FAANG in the early 2010s, just as NNs were becoming popular. Everyone on the ML research team had come to prominence before NNs, so they all specialized in different areas of ML. Nearly every researchers there treated NNs like some fad that would pass and thought they wouldn't be much better than the current techniques. How wrong they were... Of course, they're all working on NNs now.

drewcoo 4y ago

LeCun confuses the idea invention with that of innovation. Revolutions don't happen because suddenly there's a new idea, but because of one that catches on. https://www.wired.com/insights/2015/01/innovation-vs-invention/ https://www.wired.com/insights/2015/01/innovation-vs-inventi... I think he also confuses his jealousy with insight.

deleted 4y ago

[deleted]

tonymet 4y ago

In it's current iteration ChatGPT is more overhyped than crypto , and that says a lot

__seka__ 4y ago

Sometimes the innovation lies within the simple fact of how well known things are combined in order to create (good) products. "Not particularly innovative" is not really of concern. Besides OpenAI most likely received the media attention they wanted. Looking forward to the products Meta is going to release soon.

dayeye2006 4y ago

He thinks the eng work behind that makes the model training and inference possible is not so impressive as well...

andrewdubinsky 4y ago

They talked about ChatGPT by name in almost every one of my kids classes. Deep whatever may be further ahead, but no one knows the name of that thing. Everyone else is playing catch up now.

seydor 4y ago

It seems he s trying to say something that is technically correct, but exactly at the wrong time. It s going to be a lot of backlash if you go against the hype wave that we are riding. Better to explain in a more nuanced way how chatGPT managed to become so popular despite being an incremental improvement of fundamental findings of the past few years. People love it because it understands language, every time and without fail. So what if it spits out lies, this thing makes people happy, which is the value at play here

boto3 4y ago

Zuck probably just massaged him: Yann we should be cool as ChatGPT.

ACV001 4y ago

I think he just switched team and now he's on the Facebook twitter marketing team.

SillyUsername 4y ago

Sounds like bitter lemons to me from a team that's getting grief from Zuck for being shit and not getting there first or having the sense to. "First they ignore you, then they laugh at you, then they fight you, then you win"

xnx 4y ago

Useful >> Innovative

georgeburdell 4y ago

Ivory tower type calls good engineering ‘not particularly innovative’

deleted 4y ago
[deleted]