4 ms·
>> It is not an interesting take to say “but GPT-3 isn’t actually intelligent!”. Everyone knows. That's not the impression I get from the discussions on HN. Th
by YeGoblynQueenne 5y ago
>> It is not an interesting take to say “but GPT-3 isn’t actually intelligent!”. Everyone knows.
That's not the impression I get from the discussions on HN. The impression I get
is that a substantial minority (if it's not an actual majority) of commenters are convinced that it's qualitatively different than earlier
language models and that it can "understand" text. The interesting thing is that
the same kinds of comments were made about GPT-2 and I guess we'll see them with
GPT-4 and so on, also.
Even what you say about GPT-3 performing "logical" tasks (by which I assume you
mean what is more often referred to as "reasoning") is an example of that kind
of overenthusiastic misunderstanding. It's a language model. It can predict the
next tokens in a sequence. It can't reason. If the next tokens happen to
randomly match what you'd expect to see from a system capable of reasoning, then
it can appear as if it's performing reasoning, but that speaks more to your
expectations than to the system's capabilities.
And in my experience it's been extremely difficult to convince people of this
and to remind them that we know exactly what GPT-3 does and there's no super
secret hidden magick to it that somehow makes it capable of doing something it's
not designed to do. OpenAI itself of course are the ones responsible for this
confusion because their published papers are full of unsubstantiated claims
about the ability of GPT-3 to do things other than predict the next tokens in a
sequence (like "learning the rules of arithmetic" etc).
- spywaregorilla 5y agoHN shouldn't be viewed as the highest bar for collective expertise here.
- YeGoblynQueenne 5y agoWhat I mean is that there are many people commenting on HN and that makes for a large number of people who think GPT-3 is somehow "intelligent".
- spywaregorilla 5y agoSure. A lot of people believe in bigfoot though. Most of them don't matter in the space of studying primates or whatever we're calling him.
- CRConrad 5y agoThe context here was: >> >> It is not an interesting take to say “but GPT-3 isn’t actually intelligent!”. Everyone knows. GP showed that not "everyone knows" this. You're quibbling, trying to move the goalposts.
- spywaregorilla 5y agoI don't really care to debate semantics here. It felt clear to me that the context of "everyone" was the ML academic community. If you have a different take, that's fine.
- CRConrad 5y ago"Not debating semantics" is, like patriotism, the eternal refuge of the scoundrel. Everything can be reduced, in the end, to "debating semantics".
- sgt101 5y ago> If the next tokens happen to randomly match what you'd expect to see from a system capable of reasoning can we say what we would consider random and can we test to see if GPT3 does better? Has someone?
- wrinkl3 5y ago> The interesting thing is that the same kinds of comments were made about GPT-2 and I guess we'll see them with GPT-4 and so on, also. What was exciting about the GPT-2/3 was how much better they were compared to their predecessors. It's that delta that's fueling the hype among the enthusiasts, as well as the anticipation of what might happen if the delta holds for another few rounds of iteration.
- rytill 5y agoI'm one of those people, minus the straw man of it "understanding" anything. Why do you assume a language model can't learn some rudimentary reasoning from predicting the next token?