3 ms·
I think the sentences in this example are, odd vocabulary notwithstanding, fairly typical descriptions of logical implication, which I expect has non-trivial re
by joe-collins 4y ago
I think the sentences in this example are, odd vocabulary notwithstanding, fairly typical descriptions of logical implication, which I expect has non-trivial representation in the training set. GPT-3 is probably capable of matching the pattern of the tokens (sentence structure), even if the content of the tokens (spelling/vocabulary) is novel.
- uup 4y agoI doubt it. There are billions of parameters and weights all combining in non trivial ways to produce the output. I don’t think you can describe any of its workings with simple terms like “doing token substitution” even if that’s what it looks like to us when we look at the output.
- joe-collins 4y agoTry it. Reduce the tokens to single characters but keep the structure. Does it work? Maybe single characters looks too much like an abbreviation and that throws it off, because those are found in sufficiently different contexts. Exchange to two characters, three. Throw similar inputs at it and find the point where it reliably "gets it".