3 ms·

I read somewhere that unlike GPT-2, GPT-3 "understands" addition. So if you feed it "74 + 129 = " it will come up with 203, even if that specific math problem n

by m12k 6y ago

I read somewhere that unlike GPT-2, GPT-3 "understands" addition. So if you feed it "74 + 129 = " it will come up with 203, even if that specific math problem never occurred in any of the text it was trained on. That to me is the most impressive result so far - not only is it possible, but also more efficient, for the model to actually encode the rules of addition instead of just memorizing the results for specific math problems. This to me seems like the dawn of insight and understanding, and I'm super excited to see what comes next.

segfaultbuserr 6y ago
It's said that GPT-3 shows rudimentary hints of "general intelligence"-like behaviors, it can really "solve" (not just memorizing answers) many puzzles without fine-tuning it by domain-specific training data, although the performance is limited without doing it. David Chalmers [0] wrote, > When I was a graduate student in Douglas Hofstadter’s AI lab, we used letterstring analogy puzzles (if abc goes to abd, what does iijjkk go to?) as a testbed for intelligence. My fellow student Melanie Mitchell devised a program, Copycat, that was quite good at solving these puzzles. Copycat took years to write. Now Mitchell has tested GPT-3 on the same puzzles, and has found that it does a reasonable job on them (e.g. giving the answer iijjll). It is not perfect by any means and not as good as Copycat, but its results are still remarkable in a program with no fine-tuning for this domain. > What fascinates me about GPT-3 is that it suggests a potential mindless path to artificial general intelligence (or AGI). GPT-3’s training is mindless. It is just analyzing statistics of language. But to do this really well, some capacities of general intelligence are needed, and GPT-3 develops glimmers of them. [0] https://dailynous.com/2020/07/30/philosophers-gpt-3 https://dailynous.com/2020/07/30/philosophers-gpt-3
- pmoriarty 6y ago
  Speaking of Douglas Hofstadter, has he said anything about GPT-3?
- colanderman 6y ago
  I've always considered language just another form of logic (a very fuzzy one of course). To say something sensical in any language requires (almost by definition) some logical consistency. Otherwise you get non-sequiturs and insane ramblings. The greater the coherency of a narrative (e.g. paragraph-to-paragraph like GPT-3, vs. just word-to-word like a Markov chain), the greater degree of consistent logical underpinnings are needed. So that a tool trained to produce coherent human writing by necessity has embedded in itself a tool for logical reasoning is not too surprising (not to say it isn't impressive!).
chillacy 6y ago
Does it actually perform addition or is the model so big that it can store 3 digit additions in a lookup table?

by m12k 6y ago

segfaultbuserr 6y ago

It's said that GPT-3 shows rudimentary hints of "general intelligence"-like behaviors, it can really "solve" (not just memorizing answers) many puzzles without fine-tuning it by domain-specific training data, although the performance is limited without doing it. David Chalmers [0] wrote, > When I was a graduate student in Douglas Hofstadter’s AI lab, we used letterstring analogy puzzles (if abc goes to abd, what does iijjkk go to?) as a testbed for intelligence. My fellow student Melanie Mitchell devised a program, Copycat, that was quite good at solving these puzzles. Copycat took years to write. Now Mitchell has tested GPT-3 on the same puzzles, and has found that it does a reasonable job on them (e.g. giving the answer iijjll). It is not perfect by any means and not as good as Copycat, but its results are still remarkable in a program with no fine-tuning for this domain. > What fascinates me about GPT-3 is that it suggests a potential mindless path to artificial general intelligence (or AGI). GPT-3’s training is mindless. It is just analyzing statistics of language. But to do this really well, some capacities of general intelligence are needed, and GPT-3 develops glimmers of them. [0] https://dailynous.com/2020/07/30/philosophers-gpt-3 https://dailynous.com/2020/07/30/philosophers-gpt-3

pmoriarty 6y ago
Speaking of Douglas Hofstadter, has he said anything about GPT-3?
colanderman 6y ago
I've always considered language just another form of logic (a very fuzzy one of course). To say something sensical in any language requires (almost by definition) some logical consistency. Otherwise you get non-sequiturs and insane ramblings. The greater the coherency of a narrative (e.g. paragraph-to-paragraph like GPT-3, vs. just word-to-word like a Markov chain), the greater degree of consistent logical underpinnings are needed. So that a tool trained to produce coherent human writing by necessity has embedded in itself a tool for logical reasoning is not too surprising (not to say it isn't impressive!).

chillacy 6y ago

Does it actually perform addition or is the model so big that it can store 3 digit additions in a lookup table?