3 ms·

Probably not valid. With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys. The weights powering the

by phire 4y ago

Probably not valid. With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys. The weights powering the models are only so big, there simply isn't enough memory to waste storing that type of data. Instead, it's learned the general pattern of what an API key looks like and the fact that it's high-entropy random jibberish, with no predictable pattern. And when auto-completing code, it's just going to generate a random string of characters in approximately the correct format.

avian 4y ago
> With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys. This can't be true. Copilot used to reproduce code to the inverse square root function from Quake source literally. They blocked it off by blacklisting the function name. Given that, why would learning API keys be impossible? [1] https://news.ycombinator.com/item?id=27710287 https://news.ycombinator.com/item?id=27710287 [2] https://twitter.com/moyix/status/1433261377125326851 https://twitter.com/moyix/status/1433261377125326851
- sdflhasjd 4y ago
  Because the invsqrt code is well known, and exists copied in multiple places.
- phire 4y ago
  First, the inverse square root function is not high entropy. At worst it's only medium entropy. Second, it's like the complete opposite of "one-off". It's a block of code that has been copy/pasted into so many code bases, and then there have been hundreds of blog posts, articles and comments written about it. It literally has a Wikipedia article. Most of those places copy/paste the block of code verbatim. It shows up so many times in both the original gpt-3 datasets, and the extra datasets that GitHub did additional training on. That block of code is famous. No wonder why the model though those sequence of tokens were important enough to store and regurgitate verbatim with the minimal amount of prompting. But none of the same applies to an API key found in only one code base.

by phire 4y ago

avian 4y ago

> With the way machine learning networks work, they don't really learn one-off high-entropy strings like API keys. This can't be true. Copilot used to reproduce code to the inverse square root function from Quake source literally. They blocked it off by blacklisting the function name. Given that, why would learning API keys be impossible? [1] https://news.ycombinator.com/item?id=27710287 https://news.ycombinator.com/item?id=27710287 [2] https://twitter.com/moyix/status/1433261377125326851 https://twitter.com/moyix/status/1433261377125326851

sdflhasjd 4y ago
Because the invsqrt code is well known, and exists copied in multiple places.
phire 4y ago
First, the inverse square root function is not high entropy. At worst it's only medium entropy. Second, it's like the complete opposite of "one-off". It's a block of code that has been copy/pasted into so many code bases, and then there have been hundreds of blog posts, articles and comments written about it. It literally has a Wikipedia article. Most of those places copy/paste the block of code verbatim. It shows up so many times in both the original gpt-3 datasets, and the extra datasets that GitHub did additional training on. That block of code is famous. No wonder why the model though those sequence of tokens were important enough to store and regurgitate verbatim with the minimal amount of prompting. But none of the same applies to an API key found in only one code base.