3 ms·
I think the problem might be in the training data. Famous code examples are probably copied a lot and therefore appear multiple times in the training data, prom
by LeanderK 5y ago
I think the problem might be in the training data. Famous code examples are probably copied a lot and therefore appear multiple times in the training data, prompting the neural network to memorise it completely.
- avian 5y agoFamous code examples are also much more likely to be noticed. For all I know, the thing might be spewing random GPL'd code from the long tail of GitHub all the time and nobody notices because it was written by some random guy and not John Carmack.
- LeanderK 5y agoWell, it's sure speculation on my part what the root cause is, but i think OpenAI is already trying to ensure the network generalises. It's just common behaviour for neural network to memorise frequent samples, so I think my guess is quite realistic. I don't think OpenAI would not notice large-scale memorisation in their model. But as long as they don't publish more details it's just guesswork. Just keep in mind that it's a statistical tool. You can't really formally prove that it won't memorise, but I think with enough work you can get it unlikely enough that it won't matter. It's their first iteration.
- visarga 5y agoHash 10-grams and make a bloom filter. It will not generate more than 10 GPLed tokens from a source.
- mkl 5y agoCarmack didn't write this: https://en.wikipedia.org/wiki/Fast_inverse_square_root https://en.wikipedia.org/wiki/Fast_inverse_square_root
- cblconfederate 5y agoIs this function used verbatim in multiple projects? I know it's famous but how often does one use an approximation of inverse sqrt instead of the readily available cpu call in the past 20 years
- deckard1 5y agoAlso the Pareto principle. 80% of code is shit that you don't want to copy. The vast majority of github is awful hacks and insecure code that should not be touched with a ten foot pole.