3 ms·

I mean in that sense, GPT probably just extracts low-n counts as separate concepts. I'd love to see an architecture that can keep a separate short-term memory

I mean in that sense, GPT probably just extracts low-n counts as separate concepts. I'd love to see an architecture that can keep a separate short-term memory to allow it to count with multiple digits and follow algorithms. On the other hand, given what we've seen from GPT, at that point I would actually worry about it becoming a general intelligence...

nl 6y ago
> low-n counts as separate concepts But how would that work? I agree it probably doesn't "understand" math, but it has learned that number words can substitute for each other in a sentence (three ships/four ships/five ships) which isn't surprising. But it has somehow learned to link that word with the correct length of the sequence of names, which is astonishing. I can't think of obvious "cheats" that make this work. The best I can think of is that is has learned to count commas when they are separated by words.