4 ms·

Isn't the entire point of OpenAI's claim that GPT-3 is a few-shot learner that it generalizes concepts, not just syntax?

• nl 6y ago
  Yes, and that's very impressive. But to me that isn't as surprising. Not claiming I would have thought of it, but if you have a very large multi-dimensional space (such as GPT-3) then giving it some examples of something pushes it into that general area of the space. Generalizing concepts isn't a new thing - one could argue that word2vec from 2014 did that pretty well. GPT-3's "concepts" are vastly more complex than the single word (or maybe 2 word) concepts in Word2Vec though.
  • FeepingCreature 6y ago
    I mean in that sense, GPT probably just extracts low-n counts as separate concepts. I'd love to see an architecture that can keep a separate short-term memory to allow it to count with multiple digits and follow algorithms. On the other hand, given what we've seen from GPT, at that point I would actually worry about it becoming a general intelligence...
    • nl 6y ago
      > low-n counts as separate concepts But how would that work? I agree it probably doesn't "understand" math, but it has learned that number words can substitute for each other in a sentence (three ships/four ships/five ships) which isn't surprising. But it has somehow learned to link that word with the correct length of the sequence of names, which is astonishing. I can't think of obvious "cheats" that make this work. The best I can think of is that is has learned to count commas when they are separated by words.