4 ms·

In the ideal case the next token is determined by the local context (the prefix string) and the entire corpus of trained code. In this case the prefix string ha

In the ideal case the next token is determined by the local context (the prefix string) and the entire corpus of trained code. In this case the prefix string has not been seen before and so the generator must do some interpretation/extrapolation to determine the likely continuation. But in some cases, perhaps many cases, the prefix string has been seen before, or is similar enough to what has been seen before, that the best continuation is just to spit out the similar string in the training corpus. Presumably such cases can be detected due to specific patterns of activation in the network and attributions can be captured/applied. One dumb way to do this would be to include self-attributions directly in the stream of training data. So in the cases where the best continuation is to just transcribe the training data, the attributions is included in the data itself.

wrs 4y ago
That “presumably” is exactly what I’m trying to get past. How would it actually work? Once the input corpus has been chewed up and reduced to tokens, is there actually a representation of “the similar string in the corpus” any more? I don’t think people are even clear on the problem statement here. If I fed in three very similar functions from different sources, and I got a fourth, also very similar, output, what is the “traced attribution” supposed to be?
- hackinthebochs 4y ago
  There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities of the training corpus as a whole and so will not match any input data past a few highly informative tokens (ie ignoring structural tokens). The computational dynamic of doing synthesis should be noticeably different from doing verbatim copying. For one, the statistically relevant context for doing synthesis is much larger than doing verbatim copying. The prior 5 tokens may be sufficient to determine that the continuation should be the verbatim copy (eg quake's fast inverse square root). This difference must be represented in the network activation patterns in some form or another. I don't know what this difference is, but I know it's there. The case of genuine synthesis is the best case for code generators and aren't the cases where people are expecting attribution.
  - wrs 4y ago
    I see, so the hypothesis is there's some way of determining "how much choice" it has for each output token. The less choice it has, the more we will call what it's doing "verbatim copying". Once the threshold is crossed and you want to "trace" it, though, I'm still not clear on how you can map back through the latent space and figure out where in the corpus it's copying from. Other than the technique suggested upthread, where you just literally search the corpus as a separate operation.

wrs 4y ago

That “presumably” is exactly what I’m trying to get past. How would it actually work? Once the input corpus has been chewed up and reduced to tokens, is there actually a representation of “the similar string in the corpus” any more? I don’t think people are even clear on the problem statement here. If I fed in three very similar functions from different sources, and I got a fourth, also very similar, output, what is the “traced attribution” supposed to be?

hackinthebochs 4y ago
There's a clear difference between an output string that matches 10's of tokens from training data and one that is synthesized from the statistical regularities of the training corpus as a whole and so will not match any input data past a few highly informative tokens (ie ignoring structural tokens). The computational dynamic of doing synthesis should be noticeably different from doing verbatim copying. For one, the statistically relevant context for doing synthesis is much larger than doing verbatim copying. The prior 5 tokens may be sufficient to determine that the continuation should be the verbatim copy (eg quake's fast inverse square root). This difference must be represented in the network activation patterns in some form or another. I don't know what this difference is, but I know it's there. The case of genuine synthesis is the best case for code generators and aren't the cases where people are expecting attribution.
- wrs 4y ago
  I see, so the hypothesis is there's some way of determining "how much choice" it has for each output token. The less choice it has, the more we will call what it's doing "verbatim copying". Once the threshold is crossed and you want to "trace" it, though, I'm still not clear on how you can map back through the latent space and figure out where in the corpus it's copying from. Other than the technique suggested upthread, where you just literally search the corpus as a separate operation.