4 ms·

I noticed a very peculiar thing with copilot. I was writing a twitter api wrapper and had hardcoded access token for a test user. Copilot when testing a functi

I noticed a very peculiar thing with copilot. I was writing a twitter api wrapper and had hardcoded access token for a test user. Copilot when testing a function that requires a user id suggested one. I searched the id and it belonged to my test user. I searched my entire codebase but couldn't find any place I had used the particular id. The only place it could have extracted it from would have been access token which has user id as part of the string(which I hadn't noticed before this). Either this is a common code pattern or I don't know how to process this.

kall 5y ago
If it‘s always in the same location inside the api key, it seems very reasonable that it would pick up that pattern, as most projects that include a hardcoded key would include it. API key variables are probably named almost the same everywhere.
- harshitaneja 5y ago
  This is my guess as well. But I doubt anyone would be writing their code like "${userId}-${otherPart}" so copilot itself noticed that in all codebases with hardcoded twitter keys(which shouldn't be common on github I think) that their is a partial match and this is a useful information(given that in such a large corpus of all public github code partial matches would be quite common and ratio of signal to noise should be less). Whatever the case, I started as a skeptic thinking this is pure gimmick and now each day I am impressed by something new that copilot can do.
  - kall 5y ago
    If it‘s in the same position in terms of character offset, it might not make a difference to the NN. I‘m impressed as well.
bencollier49 5y ago
What was the user id? testuser1? Or something more convoluted?
- harshitaneja 5y ago
  Twitter user id for a user I created(so nothing popular as well). A 64 bit unsigned integer as string here.
arawde 5y ago
This is something I've seen with copilot with market data. I was creating a unit test in a Go codebase and I had dumped the JSON that I was going to be decoding at the top of the file, and when I started writing the assertions, Copilot was very quick to use the data from the JSON, with quite high accuracy, based solely on me typing which ticker I was going to assert against.
- harshitaneja 5y ago
  I have seen that as well and it is impressive. But this is more than that. The code contained a string variable named accessToken with a string like "218276172612672-jash127hg27128h'(random data here, not the actual id), where 218276172612672 was the user id. When testing a function that required user id it not only suggested 218276172612672 but also did it with the full context participant.follow({twitterUserId:"218276172612672"})
  - kevinsundar 5y ago
    Seems totally reasonable to me too. It probably has just seen the pattern ``` str = "{{SOME_ID_HERE}}-jash127hg27128h" participant.follow({twitterUserId:"{{SOME_ID_HERE}}"}) ```
    - harshitaneja 5y ago
      This doesn't seem likely. No one would be generating it this way as access token is issued after oauth and I am unaware of any method to get the second half of the token without the first half. And given that in the same response that contains access token, user id is passed as well so there is no need to extract it from there.
      - a-dub 5y ago
        hm. how many large integer literals are there in your code? it could just be learning that user ids are long strings of digits and is making a guess as to which long string of digits (based on some context, like sharing a line with "id" in it) might be the right one...

kall 5y ago

If it‘s always in the same location inside the api key, it seems very reasonable that it would pick up that pattern, as most projects that include a hardcoded key would include it. API key variables are probably named almost the same everywhere.

harshitaneja 5y ago
This is my guess as well. But I doubt anyone would be writing their code like "${userId}-${otherPart}" so copilot itself noticed that in all codebases with hardcoded twitter keys(which shouldn't be common on github I think) that their is a partial match and this is a useful information(given that in such a large corpus of all public github code partial matches would be quite common and ratio of signal to noise should be less). Whatever the case, I started as a skeptic thinking this is pure gimmick and now each day I am impressed by something new that copilot can do.
- kall 5y ago
  If it‘s in the same position in terms of character offset, it might not make a difference to the NN. I‘m impressed as well.

bencollier49 5y ago

What was the user id? testuser1? Or something more convoluted?

harshitaneja 5y ago
Twitter user id for a user I created(so nothing popular as well). A 64 bit unsigned integer as string here.

arawde 5y ago

This is something I've seen with copilot with market data. I was creating a unit test in a Go codebase and I had dumped the JSON that I was going to be decoding at the top of the file, and when I started writing the assertions, Copilot was very quick to use the data from the JSON, with quite high accuracy, based solely on me typing which ticker I was going to assert against.

harshitaneja 5y ago
I have seen that as well and it is impressive. But this is more than that. The code contained a string variable named accessToken with a string like "218276172612672-jash127hg27128h'(random data here, not the actual id), where 218276172612672 was the user id. When testing a function that required user id it not only suggested 218276172612672 but also did it with the full context participant.follow({twitterUserId:"218276172612672"})
- kevinsundar 5y ago
  Seems totally reasonable to me too. It probably has just seen the pattern ``` str = "{{SOME_ID_HERE}}-jash127hg27128h" participant.follow({twitterUserId:"{{SOME_ID_HERE}}"}) ```
  - harshitaneja 5y ago
    This doesn't seem likely. No one would be generating it this way as access token is issued after oauth and I am unaware of any method to get the second half of the token without the first half. And given that in the same response that contains access token, user id is passed as well so there is no need to extract it from there.
    - a-dub 5y ago
      hm. how many large integer literals are there in your code? it could just be learning that user ids are long strings of digits and is making a guess as to which long string of digits (based on some context, like sharing a line with "id" in it) might be the right one...