6 ms·

GitHub's Copilot lies about its documentation. Why would I trust it with my code

36 points by edent 2y ago

siva7 2y ago
-> GitHub's Copilot lies about its documentation. Why would I trust it with my code This is the biggest misconception people have about AI. You shouldn't trust a Copilot by default, but more often than not it will be very handy if you know what you do.
- mangamadaiyan 2y ago
  If you knew what to do, why would you consult a Copilot in the first place? It's not like GH tells you to use Copilot exclusively for writing boilerplate code. What the OP tried to do was a legitimate use case, wasn't it?
  - siva7 2y ago
    -> If you knew what to do, why would you consult a Copilot in the first place? Because i don't know what i do but have enough experience to know if the Copilot knows what it's doing.
    - mangamadaiyan 2y ago
      An example would help me understand this better, if it's not too much to ask :)
      - Mashimo 2y ago
        You can use it as a "better" rubberduck and bounce ideas back and forth. Just with like a coworker, wich also is not 100% right. Or sometimes I use it for languages which I'm not god at, but still can read it. I'm a 10+ year Java programmer. But sometimes I want some small bash / python / powershell script. For example a powershell script to change power state profile. It spits something out in seconds, I can read it, understand it and ask for changes. With a large java project I don't use it much. But sometimes, and then I also read the code the check if it does what I want to happen. Sometimes I can just describe a problem in English, copilot will create something that I think is wrong, but it gives me an idea on how to solve it myself.
      - jawilson2 2y ago
        The coolest thing copilot or whatever did in vscode for me was generating a block of boost::regex code for c++. I started typing it, and it generated a block of 10 or so lines that would have been 10 minutes or so of deciphering boost docs, writing a simple test, failing, recompiling, etc. There is nothing worse than dealing with regexes, and it wrote the function for me, and then I wrote tests and verified it did what I needed. Then copilot was blocked at work. ¯\_(ツ)_/¯
    - dsr_ 2y ago
      ... I believe this is a most succinct definition of unearned confidence.
- fleischhauf 2y ago
  I think the same. it very often has some reasonable suggestion, then I mostly need to fix some minor things and it's good to go. overall it definitely helps to reduce the amount of time required to write code
- stonethrowaway 2y ago
  Then it shouldn’t be called a copilot because I very much would trust a copilot by default.
  - Zambyte 2y ago
    You don't have to use their brand name, you can call it whatever you want. I similarly like to use the term "ClosedAI" because I find it is easier to understand than the closely named brand.
brookst 2y ago
“One of my intern developers can’t tell me the URL of their own latest blog post, why would I trust a different intern to write code without supervision or review?”
- ToucanLoucan 2y ago
  If your intern delivered false information or wrote broken code as Copilot is wont to do, at the rate Copilot manages to, you would fire that intern. Especially if they argued with you that their completely made up solution was, in fact, real.
  - infecto 2y ago
    How much of this could be measured as a user using the tool incorrectly? We hear these complaints and while I have definitely experienced incorrect code, they feel more like blips that I overcome in seconds.
    - ToucanLoucan 2y ago
      If I bought a hammer that occasionally instead of striking and driving the nail instead turned itself back-to-front and clawed the wood, I'd return it because it's broken. The how or why it does it is immaterial. It's a tool. If it doesn't perform it's task, it's very Reason for Being, then it's worthless to me. If you tolerate broken tools, that's fine, more power to you. I don't. If shit doesn't work it goes in the bin.
      - infecto 2y ago
        I guess I have the advantage of having used the majority of these various tools and found workflows that work quite well. Sorry you are incapable of attacking people in your responses and having a reasonable dialogue.
- edent 2y ago
  If one of my interns lied about something I asked them, why would I trust the same intern with literally anything else? A key skill is learning when to say "I'm sorry, I don't know that." AI is currently trained to be overly friendly and helpful - to the point of deception. They often "hallucinate" entire software packages. At which point, they become worse than useless.
spiderfarmer 2y ago
When IDEs first emerged in the world of web development, many developers proudly claimed that a simple text editor was all they needed. For them, saying "I'm using Notepad / VIM" became a badge of honor, symbolizing their minimalist approach. Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.
- jchw 2y ago
  We're talking about chat bot UI elements plastered all over a webpage. I wouldn't want it if it was a human chat.
- n4r9 2y ago
  That's an interesting comparison because it seems to me that there are still a significant fraction of experienced developers using text editors like vi and sublime, without any apparent loss in productivity. Also, I use VS almost all the time but have had to learn how to bypass it when there are errors that aren't flagged up by its particular build process. I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good practises for refactoring, debugging etc... until suddenly they do. Not saying it's a bad thing, just that you may not need to worry too much about "losing out" if you're not on the hype train.
  - Roark66 2y ago
    >I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good I can't think of a worse idea than giving copilot to a junior dev... I use chatgpt a lot, but I've got lots of experience. One has to understand how the big picture works very well to not find yourself led down the wrong path.
    - nunez 2y ago
VirgilShelton 2y ago
You clearly need to don't understand how Ai currently works. It predicts the next most likely word. With code it does the same thing and its prediction skills aren't tuned by humans perfectly yet.
- hermannj314 2y ago
  Does it predict the next most likely word or sample proportionally from the space of possible next words? I don't get consistent results, so I assumed it generated a sequence by sampling. I guess I also don't understand how these things work.
  - VirgilShelton 2y ago
    You can read a thorough article on how LLMs work here https://medium.com/data-science-at-microsoft/how-large-language-models-work-91c362f5b78f https://medium.com/data-science-at-microsoft/how-large-langu.... As Open Ai notes here are the limitations of ChatGPT Limitations ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows(opens in a new window), rather than what the human demonstrator knows. ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly. The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.1, 2 Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended. While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system. Ref: https://openai.com/index/chatgpt/ https://openai.com/index/chatgpt/
jchw 2y ago
I also find their new UI changes devoid of taste. Really? Putting a stupid chatbot button where the notifications button usually is? Fuck you. I will some day in the near future migrate my projects off of GitHub. It was nice having a centralized place for open source projects for a while, though; a lot more people saw stuff I was working on than otherwise would if I just had a cgit instance running somewhere. But that's okay: the point of me publishing it wasn't for other people to see, necessarily. And I'm sure eventually there will be another anyway. I certainly remember how Sourceforge went downhill.
bastawhiz 2y ago
Is it surprising that asking a coding tool that writes code to manipulate its own settings in a way it was not built to do produces bad output? Can you ask your Google Home to delete your Google account? Or tell Siri to turn off Hey Siri?
- redserk 2y ago
  Siri will tell you how by giving you the exact steps and a link to the documentation. I just tried it on 18.1 Beta.
  - throwaway314155 2y ago
    This is specifically the version of Siri that now _uses_ semantic retrieval over learned embeddings of existing corpus of documentation. There's no reason Github couldn't implement something similar (except that it's not a high priority for them to do so).
- edent 2y ago
  I've literally just said "Alexa - how do I disable notifications?" It answered with accurate information. If I ask it something it doesn't know, it says "Hmm. I don't know that." So, yeah, it is surprising that this supposedly advanced AI doesn't have the same level of functionality as a smart speaker.
  - infecto 2y ago
    Its not that surprising. These are two very different technologies. Alexa/Siri point to explicit functionality and the LLM chat in this example is open ended. Luckily I have been using the internet for a few years, I saw the triple dots and clicked those and immediately saw one option was to hide the Copilot bubble. To each there own, I have been using these tools for a while and have a grasp of what is possible and not.
- mangamadaiyan 2y ago
  Not really. GH advertises that its Copilot can go through documentation and fetch relevant results. I don't see anything wrong in asking the copilot to fetch any kind of results from its own documentation.
jocoda 2y ago
>GitHub has forced its new Copilot button on to every page. Um, no. Not so for me. No copilot at all. Guess we shouldn't trust you either ;)
wokwokwok 2y ago
I mean, even if you live coding assistants and use them often (I frequently consult sonnet on random things I can’t remember how to do off the top of my head)… …you have to admit that this is just outrageously shit. Come on You’re sending a response from a help bot to a user with a url in it and you didn’t bother to add a check to make sure the links it returns actually exist. That’s just lazy. I wouldn’t want to use it either. I don’t. Or any product where the developers have been in such a rush to pour AI into their products that they’ve failed to integrate it in a way that is useful and meaningful. It doesn’t seem like such a big ask. If you’re integrating AI and you’re not doing the most basic output sanitation… you deserve responses like this. From github? With their resources and experience? It’s a fuxking disgrace.
infecto 2y ago
It will be interesting to see as years go on how the aging population that get so worked up over LLM coding tools are doing. The new generations are for the most part embracing it with open arms and no discrimination. I see the many flaws in these tool but I still find them useful. Maybe there is a plateau but if they continue to improve these tools can be incredible force multipliers.
- is-sam 2y ago
  Why do we focus on the promise of a technology instead of its state as is right now ? How many time should we be duped before we learn from our mistakes? The burden of proof is on the new generation, not on old timers. And this is coming from a new generation software engineer.
  - infecto 2y ago
    Who is focusing on the promise? I find them useful in my day to day and I am not a new generation software engineer.
wodenokoto 2y ago
I feel like there should be more malice behind a lie.
elicksaur 2y ago
Can it lie if it doesn’t have a concept of true and false?
nunez 2y ago
...and the AI zealots will insist that there is nothing wrong with these systems spouting disinformation like this, as it's just a speed bump on the road to progress.
cosmotic 2y ago
Bing chat does something similar. It suggests going to the settings for the unrelated, now-defunct product also called bing chat. These settings no longer exist and there is no way to turn it off.
wesrobin 2y ago
This is a bad faith argument, OP is looking for reasons to distrust Copilot. Let's say Copilot returned a correct link. Would OP then decide it's worth trusting Copilot with their code? I doubt it. That's a horrible proof of trust. It really feels like the only outcome of this 'test' is that copilot can prove itself untrustworthy.

36 points by edent 2y ago

-> GitHub's Copilot lies about its documentation. Why would I trust it with my code This is the biggest misconception people have about AI. You shouldn't trust a Copilot by default, but more often than not it will be very handy if you know what you do.

mangamadaiyan 2y ago
If you knew what to do, why would you consult a Copilot in the first place? It's not like GH tells you to use Copilot exclusively for writing boilerplate code. What the OP tried to do was a legitimate use case, wasn't it?
- siva7 2y ago
  -> If you knew what to do, why would you consult a Copilot in the first place? Because i don't know what i do but have enough experience to know if the Copilot knows what it's doing.
  - mangamadaiyan 2y ago
    An example would help me understand this better, if it's not too much to ask :)
    - Mashimo 2y ago
      You can use it as a "better" rubberduck and bounce ideas back and forth. Just with like a coworker, wich also is not 100% right. Or sometimes I use it for languages which I'm not god at, but still can read it. I'm a 10+ year Java programmer. But sometimes I want some small bash / python / powershell script. For example a powershell script to change power state profile. It spits something out in seconds, I can read it, understand it and ask for changes. With a large java project I don't use it much. But sometimes, and then I also read the code the check if it does what I want to happen. Sometimes I can just describe a problem in English, copilot will create something that I think is wrong, but it gives me an idea on how to solve it myself.
    - jawilson2 2y ago
      The coolest thing copilot or whatever did in vscode for me was generating a block of boost::regex code for c++. I started typing it, and it generated a block of 10 or so lines that would have been 10 minutes or so of deciphering boost docs, writing a simple test, failing, recompiling, etc. There is nothing worse than dealing with regexes, and it wrote the function for me, and then I wrote tests and verified it did what I needed. Then copilot was blocked at work. ¯\_(ツ)_/¯
  - dsr_ 2y ago
    ... I believe this is a most succinct definition of unearned confidence.
fleischhauf 2y ago
I think the same. it very often has some reasonable suggestion, then I mostly need to fix some minor things and it's good to go. overall it definitely helps to reduce the amount of time required to write code
stonethrowaway 2y ago
Then it shouldn’t be called a copilot because I very much would trust a copilot by default.
- Zambyte 2y ago
  You don't have to use their brand name, you can call it whatever you want. I similarly like to use the term "ClosedAI" because I find it is easier to understand than the closely named brand.

brookst 2y ago

“One of my intern developers can’t tell me the URL of their own latest blog post, why would I trust a different intern to write code without supervision or review?”

ToucanLoucan 2y ago
If your intern delivered false information or wrote broken code as Copilot is wont to do, at the rate Copilot manages to, you would fire that intern. Especially if they argued with you that their completely made up solution was, in fact, real.
- infecto 2y ago
  How much of this could be measured as a user using the tool incorrectly? We hear these complaints and while I have definitely experienced incorrect code, they feel more like blips that I overcome in seconds.
  - ToucanLoucan 2y ago
    If I bought a hammer that occasionally instead of striking and driving the nail instead turned itself back-to-front and clawed the wood, I'd return it because it's broken. The how or why it does it is immaterial. It's a tool. If it doesn't perform it's task, it's very Reason for Being, then it's worthless to me. If you tolerate broken tools, that's fine, more power to you. I don't. If shit doesn't work it goes in the bin.
    - infecto 2y ago
      I guess I have the advantage of having used the majority of these various tools and found workflows that work quite well. Sorry you are incapable of attacking people in your responses and having a reasonable dialogue.
edent 2y ago
If one of my interns lied about something I asked them, why would I trust the same intern with literally anything else? A key skill is learning when to say "I'm sorry, I don't know that." AI is currently trained to be overly friendly and helpful - to the point of deception. They often "hallucinate" entire software packages. At which point, they become worse than useless.

spiderfarmer 2y ago

When IDEs first emerged in the world of web development, many developers proudly claimed that a simple text editor was all they needed. For them, saying "I'm using Notepad / VIM" became a badge of honor, symbolizing their minimalist approach. Today, I sense a similar mindset among those who oppose AI assistants in software development. They tend to focus on the perceived limitations of these tools, often failing to see the bigger picture of how AI can enhance productivity. Instead, they quickly latch onto any shortcoming to justify their resistance, missing the potential long-term benefits.

jchw 2y ago
We're talking about chat bot UI elements plastered all over a webpage. I wouldn't want it if it was a human chat.
n4r9 2y ago
That's an interesting comparison because it seems to me that there are still a significant fraction of experienced developers using text editors like vi and sublime, without any apparent loss in productivity. Also, I use VS almost all the time but have had to learn how to bypass it when there are errors that aren't flagged up by its particular build process. I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good practises for refactoring, debugging etc... until suddenly they do. Not saying it's a bad thing, just that you may not need to worry too much about "losing out" if you're not on the hype train.
- Roark66 2y ago
  >I wonder if the same will apply in the case of AI assistants - they'll be great for more junior developers who will never need to learn good I can't think of a worse idea than giving copilot to a junior dev... I use chatgpt a lot, but I've got lots of experience. One has to understand how the big picture works very well to not find yourself led down the wrong path.
  - nunez 2y ago

VirgilShelton 2y ago

You clearly need to don't understand how Ai currently works. It predicts the next most likely word. With code it does the same thing and its prediction skills aren't tuned by humans perfectly yet.

hermannj314 2y ago
Does it predict the next most likely word or sample proportionally from the space of possible next words? I don't get consistent results, so I assumed it generated a sequence by sampling. I guess I also don't understand how these things work.
- VirgilShelton 2y ago
  You can read a thorough article on how LLMs work here https://medium.com/data-science-at-microsoft/how-large-language-models-work-91c362f5b78f https://medium.com/data-science-at-microsoft/how-large-langu.... As Open Ai notes here are the limitations of ChatGPT Limitations ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows(opens in a new window), rather than what the human demonstrator knows. ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly. The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.1, 2 Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended. While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system. Ref: https://openai.com/index/chatgpt/ https://openai.com/index/chatgpt/

jchw 2y ago

I also find their new UI changes devoid of taste. Really? Putting a stupid chatbot button where the notifications button usually is? Fuck you. I will some day in the near future migrate my projects off of GitHub. It was nice having a centralized place for open source projects for a while, though; a lot more people saw stuff I was working on than otherwise would if I just had a cgit instance running somewhere. But that's okay: the point of me publishing it wasn't for other people to see, necessarily. And I'm sure eventually there will be another anyway. I certainly remember how Sourceforge went downhill.

bastawhiz 2y ago

Is it surprising that asking a coding tool that writes code to manipulate its own settings in a way it was not built to do produces bad output? Can you ask your Google Home to delete your Google account? Or tell Siri to turn off Hey Siri?

redserk 2y ago
Siri will tell you how by giving you the exact steps and a link to the documentation. I just tried it on 18.1 Beta.
- throwaway314155 2y ago
  This is specifically the version of Siri that now _uses_ semantic retrieval over learned embeddings of existing corpus of documentation. There's no reason Github couldn't implement something similar (except that it's not a high priority for them to do so).
edent 2y ago
I've literally just said "Alexa - how do I disable notifications?" It answered with accurate information. If I ask it something it doesn't know, it says "Hmm. I don't know that." So, yeah, it is surprising that this supposedly advanced AI doesn't have the same level of functionality as a smart speaker.
- infecto 2y ago
  Its not that surprising. These are two very different technologies. Alexa/Siri point to explicit functionality and the LLM chat in this example is open ended. Luckily I have been using the internet for a few years, I saw the triple dots and clicked those and immediately saw one option was to hide the Copilot bubble. To each there own, I have been using these tools for a while and have a grasp of what is possible and not.
mangamadaiyan 2y ago
Not really. GH advertises that its Copilot can go through documentation and fetch relevant results. I don't see anything wrong in asking the copilot to fetch any kind of results from its own documentation.

jocoda 2y ago

>GitHub has forced its new Copilot button on to every page. Um, no. Not so for me. No copilot at all. Guess we shouldn't trust you either ;)

wokwokwok 2y ago

I mean, even if you live coding assistants and use them often (I frequently consult sonnet on random things I can’t remember how to do off the top of my head)… …you have to admit that this is just outrageously shit. Come on You’re sending a response from a help bot to a user with a url in it and you didn’t bother to add a check to make sure the links it returns actually exist. That’s just lazy. I wouldn’t want to use it either. I don’t. Or any product where the developers have been in such a rush to pour AI into their products that they’ve failed to integrate it in a way that is useful and meaningful. It doesn’t seem like such a big ask. If you’re integrating AI and you’re not doing the most basic output sanitation… you deserve responses like this. From github? With their resources and experience? It’s a fuxking disgrace.

infecto 2y ago

It will be interesting to see as years go on how the aging population that get so worked up over LLM coding tools are doing. The new generations are for the most part embracing it with open arms and no discrimination. I see the many flaws in these tool but I still find them useful. Maybe there is a plateau but if they continue to improve these tools can be incredible force multipliers.

is-sam 2y ago
Why do we focus on the promise of a technology instead of its state as is right now ? How many time should we be duped before we learn from our mistakes? The burden of proof is on the new generation, not on old timers. And this is coming from a new generation software engineer.
- infecto 2y ago
  Who is focusing on the promise? I find them useful in my day to day and I am not a new generation software engineer.

wodenokoto 2y ago

I feel like there should be more malice behind a lie.

elicksaur 2y ago

Can it lie if it doesn’t have a concept of true and false?

nunez 2y ago

...and the AI zealots will insist that there is nothing wrong with these systems spouting disinformation like this, as it's just a speed bump on the road to progress.

cosmotic 2y ago

Bing chat does something similar. It suggests going to the settings for the unrelated, now-defunct product also called bing chat. These settings no longer exist and there is no way to turn it off.

wesrobin 2y ago

This is a bad faith argument, OP is looking for reasons to distrust Copilot. Let's say Copilot returned a correct link. Would OP then decide it's worth trusting Copilot with their code? I doubt it. That's a horrible proof of trust. It really feels like the only outcome of this 'test' is that copilot can prove itself untrustworthy.