5 ms·

Wow. Seriously? It can make an inference like that? I wonder if “optimystic” shows up at all in the training data or if this was purely from some ability to de

Wow. Seriously? It can make an inference like that? I wonder if “optimystic” shows up at all in the training data or if this was purely from some ability to detect those two source words.

lionkor 3y ago
Its the problem with fully proprietary AI like this: You cannot prove that this question and this answer wasnt in the training set, so you cannot argue for its ability to infer or reason.
- 20after4 3y ago
  You can't prove that they aren't answering ChatGPT questions with real humans, either.
  - lionkor 3y ago
    You're making my point for me. Exactly, a fully closed source language model cannot be evaluated because there is no way to know why it replies the way it does. My point exactly.
    - 20after4 3y ago
      Yeah, I agree, and I didn't really mean for my reply to be a rebuttal to your point at all. :)
jonahx 3y ago
Short answer: for all practical purposes, yes, it can and it does. For each specific example, there is no way to tell for sure (afaik) if the example was in the training set. But you can easily run some experiments yourself, inventing your own words which would not likely be in the training set, especially when taken together. I have done this, and GPT4 will frequently make inferences on par with the "optimystic" one. For example I just tried "surfrandma" and it said "It appears to be a combination of the words "surf" and "grandma", but without additional context, it's challenging to provide a precise meaning."
Sharlin 3y ago
It can do so much more that the fact that it can go from "optimystic" to "optimistic" and "mystic" is extremely mundane in comparison.
- Waterluvian 3y ago
  Like what? And how does one measure that it is more impressive or less mundane?
  - Sharlin 3y ago
    Like just about anything. And the measure is something like "does someone who has spent some time with GPT-4 find it at all surprising that it can do X". A posteriori, it would be much more surprising if GPT-4 failed to resolve "optimystic" to "mystic" and "optimistic". Even though it's handicapped by its encoding when it comes to wordplays.

Wow. Seriously? It can make an inference like that? I wonder if “optimystic” shows up at all in the training data or if this was purely from some ability to detect those two source words.

lionkor 3y ago

Its the problem with fully proprietary AI like this: You cannot prove that this question and this answer wasnt in the training set, so you cannot argue for its ability to infer or reason.

20after4 3y ago
You can't prove that they aren't answering ChatGPT questions with real humans, either.
- lionkor 3y ago
  You're making my point for me. Exactly, a fully closed source language model cannot be evaluated because there is no way to know why it replies the way it does. My point exactly.
  - 20after4 3y ago
    Yeah, I agree, and I didn't really mean for my reply to be a rebuttal to your point at all. :)

jonahx 3y ago

Short answer: for all practical purposes, yes, it can and it does. For each specific example, there is no way to tell for sure (afaik) if the example was in the training set. But you can easily run some experiments yourself, inventing your own words which would not likely be in the training set, especially when taken together. I have done this, and GPT4 will frequently make inferences on par with the "optimystic" one. For example I just tried "surfrandma" and it said "It appears to be a combination of the words "surf" and "grandma", but without additional context, it's challenging to provide a precise meaning."

Sharlin 3y ago

It can do so much more that the fact that it can go from "optimystic" to "optimistic" and "mystic" is extremely mundane in comparison.

Waterluvian 3y ago
Like what? And how does one measure that it is more impressive or less mundane?
- Sharlin 3y ago
  Like just about anything. And the measure is something like "does someone who has spent some time with GPT-4 find it at all surprising that it can do X". A posteriori, it would be much more surprising if GPT-4 failed to resolve "optimystic" to "mystic" and "optimistic". Even though it's handicapped by its encoding when it comes to wordplays.