3 ms·

That “presumably” is exactly what I’m trying to get past. How would it actually work? Once the input corpus has been chewed up and reduced to tokens, is there a

by wrs 4y ago

That “presumably” is exactly what I’m trying to get past. How would it actually work? Once the input corpus has been chewed up and reduced to tokens, is there actually a representation of “the similar string in the corpus” any more? I don’t think people are even clear on the problem statement here. If I fed in three very similar functions from different sources, and I got a fourth, also very similar, output, what is the “traced attribution” supposed to be?

hackinthebochs 4y ago
There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities of the training corpus as a whole and so will not match any input data past a few highly informative tokens (ie ignoring structural tokens). The computational dynamic of doing synthesis should be noticeably different from doing verbatim copying. For one, the statistically relevant context for doing synthesis is much larger than doing verbatim copying. The prior 5 tokens may be sufficient to determine that the continuation should be the verbatim copy (eg quake's fast inverse square root). This difference must be represented in the network activation patterns in some form or another. I don't know what this difference is, but I know it's there. The case of genuine synthesis is the best case for code generators and aren't the cases where people are expecting attribution.
- wrs 4y ago
  I see, so the hypothesis is there's some way of determining "how much choice" it has for each output token. The less choice it has, the more we will call what it's doing "verbatim copying". Once the threshold is crossed and you want to "trace" it, though, I'm still not clear on how you can map back through the latent space and figure out where in the corpus it's copying from. Other than the technique suggested upthread, where you just literally search the corpus as a separate operation.

by wrs 4y ago

hackinthebochs 4y ago

There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities of the training corpus as a whole and so will not match any input data past a few highly informative tokens (ie ignoring structural tokens). The computational dynamic of doing synthesis should be noticeably different from doing verbatim copying. For one, the statistically relevant context for doing synthesis is much larger than doing verbatim copying. The prior 5 tokens may be sufficient to determine that the continuation should be the verbatim copy (eg quake's fast inverse square root). This difference must be represented in the network activation patterns in some form or another. I don't know what this difference is, but I know it's there. The case of genuine synthesis is the best case for code generators and aren't the cases where people are expecting attribution.

wrs 4y ago
I see, so the hypothesis is there's some way of determining "how much choice" it has for each output token. The less choice it has, the more we will call what it's doing "verbatim copying". Once the threshold is crossed and you want to "trace" it, though, I'm still not clear on how you can map back through the latent space and figure out where in the corpus it's copying from. Other than the technique suggested upthread, where you just literally search the corpus as a separate operation.