3 ms·

see some analysis of the scope of this issue here: https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/res

by yayr 5y ago

see some analysis of the scope of this issue here: https://docs.github.com/en/github/copilot/research-recitation https://docs.github.com/en/github/copilot/research-recitatio... especially: Conclusion and Next Steps. This investigation demonstrates that GitHub Copilot can quote a body of code verbatim, but that it rarely does so, and when it does, it mostly quotes code that everybody quotes, and mostly at the beginning of a file, as if to break the ice. But there’s still one big difference between GitHub Copilot reciting code and me reciting a poem: I know when I’m quoting. I would also like to know when Copilot is echoing existing code rather than coming up with its own ideas. That way, I’m able to look up background information about that code, and to include credit where credit is due. The answer is obvious: sharing the prefiltering solution we used in this analysis to detect overlap with the training set. When a suggestion contains snippets copied from the training set, the UI should simply tell you where it’s quoted from. You can then either include proper attribution or decide against using that code altogether. This duplication search is not yet integrated into the technical preview, but we plan to do so. And we will both continue to work on decreasing rates of recitation, and on making its detection more precise.

temac 5y ago
The way "AI" works for now, Copilot never comes with its own ideas, as it is incapable of deductive reasoning. It basically just detects from the context then mixes variations of things it learned. If there is nothing to mix (that is if there is a single source), the risk of spitting verbatim is high. But if there are multiple sources and some mixing and some amount of tiny differences, said differences better not be not too trivial because I don't see why we would suddenly drop Abstraction-Filtration-Comparison approaches... So their defense of the like "oh it's fine it very rarely emits verbatim things" is bullshit anyway. That's an answer to a wrong question, at least given the answer is in this direction (would there be ton of verbatim recitation, they obviously would not try to wave away the problem like that -- however we can not conclude anything from verbatim output being rare, despite them stating that as if it a quite central and strong argument)

by yayr 5y ago

temac 5y ago

The way "AI" works for now, Copilot never comes with its own ideas, as it is incapable of deductive reasoning. It basically just detects from the context then mixes variations of things it learned. If there is nothing to mix (that is if there is a single source), the risk of spitting verbatim is high. But if there are multiple sources and some mixing and some amount of tiny differences, said differences better not be not too trivial because I don't see why we would suddenly drop Abstraction-Filtration-Comparison approaches... So their defense of the like "oh it's fine it very rarely emits verbatim things" is bullshit anyway. That's an answer to a wrong question, at least given the answer is in this direction (would there be ton of verbatim recitation, they obviously would not try to wave away the problem like that -- however we can not conclude anything from verbatim output being rare, despite them stating that as if it a quite central and strong argument)