3 ms·
There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities
by hackinthebochs 4y ago
There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities of the training corpus as a whole and so will not match any input data past a few highly informative tokens (ie ignoring structural tokens). The computational dynamic of doing synthesis should be noticeably different from doing verbatim copying. For one, the statistically relevant context for doing synthesis is much larger than doing verbatim copying. The prior 5 tokens may be sufficient to determine that the continuation should be the verbatim copy (eg quake's fast inverse square root). This difference must be represented in the network activation patterns in some form or another. I don't know what this difference is, but I know it's there.
The case of genuine synthesis is the best case for code generators and aren't the cases where people are expecting attribution.
- wrs 4y agoI see, so the hypothesis is there's some way of determining "how much choice" it has for each output token. The less choice it has, the more we will call what it's doing "verbatim copying". Once the threshold is crossed and you want to "trace" it, though, I'm still not clear on how you can map back through the latent space and figure out where in the corpus it's copying from. Other than the technique suggested upthread, where you just literally search the corpus as a separate operation.