6 ms·

It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it wen

by st1x7 6y ago

It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.

pfortuny 6y ago
That is the best summary I have read in a while. Exactly that, only with 3000 words (say) of “prediction”.
drcode 6y ago
What evidence do I have that I'm more than a fancy autocomplete, myself? The use of squishy protestations, in lieu of objective metrics, make LeCun's argument rather unconvincing.
- btrask 6y ago
  If you are nothing more than fancy autocomplete, then why should we argue with you? GPT is not trying to make a point and is not capable of changing its mind. You, hopefully, are. Edit: I don't think you should be getting downvoted because it's a valid (and interesting) question.
  - dwaltrip 6y ago
    That is an interesting point. However, one can potentially conceive of us human changing our minds as a form of real-time model updating.
- nightski 6y ago
  High level reasoning and planning about what you are writing about. GPT-3 generates text one word at a time. The results are impressive for what it is. But it will not plan out what it wants to say ahead of time and construct the message to achieve that objective.
  - lostmsu 6y ago
    But of course it does within its own horizon (e.g. token window).
- bmgxyz 6y ago
  I was about to write a reply claiming that you're different from autocomplete because you take input from more sources than just the words you've said before (e.g. your vision), but actually I can't see how that's much different from a language model. The approach seems the same, and all that's really different is the shape of the input data. But this uncovers difficult questions about free will. If we're all just autocompleting based on a combination of the world around us, our internal state, and the physical laws, then what even is intelligence anyway? This view reduces thought to nothing more than an interesting dust storm. Still, I find the original argument compelling, if not logically convincing. There does seem to be something missing from GPT-3 that differs fundamentally from human intelligence or AGI. But maybe that's an illusion. Edit: I don't think you should have been downvoted, since your question is valid and constructive in my view.
  - minerjoe 6y ago
    > we're all just autocompleting That's pretty much it. I do belive it's possible to actually develop "will", but almost nobody thinks that they need to work on such things. They confuse being a programmed robot with being a programmer.
vladf 6y ago
I happen to use slightly less fancy and expensive GPT-2 based autocomplete, and it's amazing. https://tabnine.com https://tabnine.com
- tmalsburg2 6y ago
  Interesting. As a reading researcher, I imagine that this could potentially introduce subtle and difficult to spot bugs when you get a proposed completion that looks about right i.e. close enough to what you imagined. Has this been an issue in your experience?
  - 0-_-0 6y ago
    I use it too and it hasn't, you still need to look at the suggestions before you accept them. It's not that good, so I won't lose my job just yet.
  - surething123 6y ago
    Not the person to whom you asked this question, but I'm also a user of TabNine. In my experience, the "recommendations" / autocompletions provided by the tool are usually very short (probably less than 20 characters on average), and I don't use it for terribly complex chunks of code. Where I like it most is in the initialization of common code chunks like `if` and `for` loops, using variables instantiated in nearby preceding lines. It figures out things like `for(customer in customers)` as I'm writing `for`.
  - vladf 6y ago
    No, it's not auto-completing function blocks, just simple expressions that are easy to validate. E.g., let lo = 0; let hi = vec.len(); let mid = lo + (hi will autocomplete to `(hi - lo) / 2` as the second autocomplete option (so I'd hit tab twice). If you were to "score" it based on top-5 it'd probably be pretty bad at guessing my intent, but then again, I get to _opt-in_ to suggestions so it just needs to be right often enough, as it doesn't bother me much to keep typing.
FatalLogic 6y ago
You're correct. It's only autocomplete on steroids. But I think it's remarkable that something with the very simple goal of autocomplete can, for a few sentences, sound almost alive
- azinman2 6y ago
  Because we as humans easily anthropomorphize. [1] [1] https://en.wikipedia.org/wiki/The_Media_Equation https://en.wikipedia.org/wiki/The_Media_Equation
  - FatalLogic 6y ago
    Yes, though our tendency to anthropomorphize also helps makes other people human
tiborsaas 6y ago
You are just a fancy and efficient autocomplete too. When you speak or write, some words have a higher probability than others. You pick alternatives, but they are limited. Of course there are more layers in the human mind, but GPT-3 is a really impressive milestone towards AGI. It's so easy to downplay every advanced tech, it's actually fun. Planes? Just a flying metal tube. Self landing rockets? Just applied physics. Smartphones? Just really good fab processes. The internet? Just a bunch of computers. CRISPR? Just a molecular scissor.
- st1x7 6y ago
  I'm not reducing GPT-3 to the extent that you're suggesting. I'm pointing out (and so does LeCun in his post) that it's a language model designed to continue a sequence of words. It has no understanding of the world and is no particularly suited for knowledge extraction or conversation. > GPT-3 is a really impressive milestone towards AGI We really don't know this. It's a big step for the field of language models, that's for sure. But we're so far from AGI that nobody knows which direction it's in and whether it exists at all.
  - moultano 6y ago
    > it's a language model designed to continue a sequence of words. If a language model were able to do this task perfectly, it would be indistinguishable from intelligence, because continuing a sequence of words requires reasoning. You cannot conclude that has no understanding based solely on what it is trained to do when the task it is trained on would be sufficient to demonstrate understanding were it to fully succeed. There are lots of reasons to be skeptical of its potential, but this isn't one of them.
    - uryga 6y ago
      > you cannot conclude that [a model] has no understanding based solely on what it is trained to do agreed. but that's not everything we're basing our conclusions on – we also know that GPT-3 was trained purely on text, and i (and presumably GP) don't think that's a path towards "understanding". in other words, i think being a language model [trained only using a text corpus] is a valid reason to be skeptical of its potential :)
corobo 6y ago
> just a fancy and expensive autocomplete It may be, but there's a lot in that fancy. If it were 'just' an autocomplete we'd all be using markov chains for our dumb chatbots like we were in the 2000s
6gvONxR4sf7o 6y ago
There’s a lot of baggage being thrown into the word fancy here. Any (and I mean any) distribution can be factored as a sequence of its random variables, with the next one being conditional on everything that’s come before, aka autocomplete. That said, I agree more closely with LeCun than the hypers here.

by st1x7 6y ago

pfortuny 6y ago

That is the best summary I have read in a while. Exactly that, only with 3000 words (say) of “prediction”.

drcode 6y ago

What evidence do I have that I'm more than a fancy autocomplete, myself? The use of squishy protestations, in lieu of objective metrics, make LeCun's argument rather unconvincing.

btrask 6y ago
If you are nothing more than fancy autocomplete, then why should we argue with you? GPT is not trying to make a point and is not capable of changing its mind. You, hopefully, are. Edit: I don't think you should be getting downvoted because it's a valid (and interesting) question.
- dwaltrip 6y ago
  That is an interesting point. However, one can potentially conceive of us human changing our minds as a form of real-time model updating.
nightski 6y ago
High level reasoning and planning about what you are writing about. GPT-3 generates text one word at a time. The results are impressive for what it is. But it will not plan out what it wants to say ahead of time and construct the message to achieve that objective.
- lostmsu 6y ago
  But of course it does within its own horizon (e.g. token window).
bmgxyz 6y ago
I was about to write a reply claiming that you're different from autocomplete because you take input from more sources than just the words you've said before (e.g. your vision), but actually I can't see how that's much different from a language model. The approach seems the same, and all that's really different is the shape of the input data. But this uncovers difficult questions about free will. If we're all just autocompleting based on a combination of the world around us, our internal state, and the physical laws, then what even is intelligence anyway? This view reduces thought to nothing more than an interesting dust storm. Still, I find the original argument compelling, if not logically convincing. There does seem to be something missing from GPT-3 that differs fundamentally from human intelligence or AGI. But maybe that's an illusion. Edit: I don't think you should have been downvoted, since your question is valid and constructive in my view.
- minerjoe 6y ago
  > we're all just autocompleting That's pretty much it. I do belive it's possible to actually develop "will", but almost nobody thinks that they need to work on such things. They confuse being a programmed robot with being a programmer.

vladf 6y ago

I happen to use slightly less fancy and expensive GPT-2 based autocomplete, and it's amazing. https://tabnine.com https://tabnine.com

tmalsburg2 6y ago
Interesting. As a reading researcher, I imagine that this could potentially introduce subtle and difficult to spot bugs when you get a proposed completion that looks about right i.e. close enough to what you imagined. Has this been an issue in your experience?
- 0-_-0 6y ago
  I use it too and it hasn't, you still need to look at the suggestions before you accept them. It's not that good, so I won't lose my job just yet.
- surething123 6y ago
  Not the person to whom you asked this question, but I'm also a user of TabNine. In my experience, the "recommendations" / autocompletions provided by the tool are usually very short (probably less than 20 characters on average), and I don't use it for terribly complex chunks of code. Where I like it most is in the initialization of common code chunks like `if` and `for` loops, using variables instantiated in nearby preceding lines. It figures out things like `for(customer in customers)` as I'm writing `for`.
- vladf 6y ago
  No, it's not auto-completing function blocks, just simple expressions that are easy to validate. E.g., let lo = 0; let hi = vec.len(); let mid = lo + (hi will autocomplete to `(hi - lo) / 2` as the second autocomplete option (so I'd hit tab twice). If you were to "score" it based on top-5 it'd probably be pretty bad at guessing my intent, but then again, I get to _opt-in_ to suggestions so it just needs to be right often enough, as it doesn't bother me much to keep typing.

FatalLogic 6y ago

You're correct. It's only autocomplete on steroids. But I think it's remarkable that something with the very simple goal of autocomplete can, for a few sentences, sound almost alive

azinman2 6y ago
Because we as humans easily anthropomorphize. [1] [1] https://en.wikipedia.org/wiki/The_Media_Equation https://en.wikipedia.org/wiki/The_Media_Equation
- FatalLogic 6y ago
  Yes, though our tendency to anthropomorphize also helps makes other people human

tiborsaas 6y ago

You are just a fancy and efficient autocomplete too. When you speak or write, some words have a higher probability than others. You pick alternatives, but they are limited. Of course there are more layers in the human mind, but GPT-3 is a really impressive milestone towards AGI. It's so easy to downplay every advanced tech, it's actually fun. Planes? Just a flying metal tube. Self landing rockets? Just applied physics. Smartphones? Just really good fab processes. The internet? Just a bunch of computers. CRISPR? Just a molecular scissor.

st1x7 6y ago
I'm not reducing GPT-3 to the extent that you're suggesting. I'm pointing out (and so does LeCun in his post) that it's a language model designed to continue a sequence of words. It has no understanding of the world and is no particularly suited for knowledge extraction or conversation. > GPT-3 is a really impressive milestone towards AGI We really don't know this. It's a big step for the field of language models, that's for sure. But we're so far from AGI that nobody knows which direction it's in and whether it exists at all.
- moultano 6y ago
  > it's a language model designed to continue a sequence of words. If a language model were able to do this task perfectly, it would be indistinguishable from intelligence, because continuing a sequence of words requires reasoning. You cannot conclude that has no understanding based solely on what it is trained to do when the task it is trained on would be sufficient to demonstrate understanding were it to fully succeed. There are lots of reasons to be skeptical of its potential, but this isn't one of them.
  - uryga 6y ago
    > you cannot conclude that [a model] has no understanding based solely on what it is trained to do agreed. but that's not everything we're basing our conclusions on – we also know that GPT-3 was trained purely on text, and i (and presumably GP) don't think that's a path towards "understanding". in other words, i think being a language model [trained only using a text corpus] is a valid reason to be skeptical of its potential :)

corobo 6y ago

> just a fancy and expensive autocomplete It may be, but there's a lot in that fancy. If it were 'just' an autocomplete we'd all be using markov chains for our dumb chatbots like we were in the 2000s

6gvONxR4sf7o 6y ago

There’s a lot of baggage being thrown into the word fancy here. Any (and I mean any) distribution can be factored as a sequence of its random variables, with the next one being conditional on everything that’s come before, aka autocomplete. That said, I agree more closely with LeCun than the hypers here.