4 ms·

I use GitHub Copilot at work. At first, I found myself blindly accepting its suggestions. They seemed so plausible! After I got bit in the behind by enough wei

by elil17 4y ago

I use GitHub Copilot at work. At first, I found myself blindly accepting its suggestions. They seemed so plausible! After I got bit in the behind by enough weird, AI generated bugs, I learned the right level of trust. I now know how to read the code it writes and check the parts it’s likely to get wrong. I think, with this generation of the technology, being too trusting of AIs is going to be an issue mainly for people who don’t interact with them much or in circumstances where the boundaries between real people and AIs are unclear (e.g. customer service chats). Once AIs’ error rate goes down, that’s where it’s going to be tricky. Humans are notoriously bad at monitoring low failure rate processes. Once things go right a hundred times, we like to assume they’ll go right forever. It’s what makes getting people to properly monitor things like nuclear power plants such a challenge.

swayvil 4y ago
First hand observation aside, the 2 ingredients of truth are authority and plausibility. Robots make great authorities. They're shiny and speak with confidence.
- elil17 4y ago
  Until you realize they’re dumb. I don’t think people are so naive that they’ll ignore clearly dumb robots that they work with every day. I do think people will make those mistakes with robots they’re not that familiar with.
  - swayvil 4y ago
    Challenge the assertions of of a shiny-confident-robot and you will be challenged right back with, according to what shiny-confident-robot? And no, first hand observation doesn't count, because that's mere anecdote.
    - elil17 4y ago
      It's an anecdote which is meant to illustrate a widely observed behavior among many humans: we're better at accounting for frequent errors than we are at accounting for rare ones. This is well established in the field of human reliability analysis (HRA). Take a look at the HRA method HEART (https://www.epd.gov.hk/eia/register/report/eiareport/eia_2242014/EIA/app/app12.10.pdf https://www.epd.gov.hk/eia/register/report/eiareport/eia_224...). There are three different error producing conditions (EPCs) related to rare failures (EPCs 1, 3, and 12 in table 1.3). As for my argument that people who frequently work with this technology aren't going to be so susceptible to "shiny-confident-robot" - that is based on my first hand observation. But the counterargument (people trust LLMs too much) is not based in data either. The Pearce, et. al paper everyone cites about security vulnerabilities didn't look at how much people trust the code, just how frequently it gives wrong answers (https://arxiv.org/pdf/2108.09293.pdf https://arxiv.org/pdf/2108.09293.pdf). An actual user study (n=20) showed that Copilot users engage critically with the tool by examining and testing suggestions, comparing multiple suggestions, and crosschecking it against documentation (https://arxiv.org/pdf/2206.15000.pdf https://arxiv.org/pdf/2206.15000.pdf).
  - fakedang 4y ago
    They're not dumb. They're management consultants!
  - jfk13 4y ago
    Experience suggests that people are all too willing to unthinkingly follow instructions just because "the computer said so". https://theweek.com/articles/464674/8-drivers-who-blindly-followed-gps-into-disaster https://theweek.com/articles/464674/8-drivers-who-blindly-fo...
    - ghaff 4y ago
      People have certainly done some dumb things in the GPS vein. That said, if you're taking confident navigation directions from either a person or a computer, it can be hard to make a split-second decision that something "doesn't seem right." (Obviously some of the more extreme cases still shouldn't happen.)
      - still_grokking 4y ago
        > People have certainly done some dumb things This sentence doesn't need to be continued. It's already the ultimate state of affairs since forever.
        quicklime 4y ago
        That's true if your only goal is to generate truthful text. But if you're trying to generate useful text, continuing the sentence is necessary. Maybe that'll be the next step once LLMs get very good at generating truthful text :)
    - elil17 4y ago
      I would say GPS is in the zone of having such a low error rate that people can't process it right. If, as an example, GitHub Copilot added a security vulnerability in one in ten thousand lines of code, that would be a disaster because no one would double check it's work. Currently Copilot adds a security vulnerability in one in ten lines of code (made up number), so people with even a tad of experience using it know not to blindly trust it. I don't mean to dismiss the issue of over-trust in computer systems - I think it's a very serious issue, and one that needs to be addressed across a lot of disciplines.
- bambax 4y ago
  Exactly. It's quite bad. In the case of ChatGPT, it could probably be mitigated by prefacing each interaction or each answer with a warning "I'm just a machine that makes stuff up; don't take what I say too seriously". But for all the talk about "benefiting humanity" and being good to the world from so-called OpenAI, what we get instead is a constant affirmation that it's never wrong, with the upbeat tone of a dystopian policeman.

by elil17 4y ago

swayvil 4y ago

First hand observation aside, the 2 ingredients of truth are authority and plausibility. Robots make great authorities. They're shiny and speak with confidence.

elil17 4y ago
Until you realize they’re dumb. I don’t think people are so naive that they’ll ignore clearly dumb robots that they work with every day. I do think people will make those mistakes with robots they’re not that familiar with.
- swayvil 4y ago
  Challenge the assertions of of a shiny-confident-robot and you will be challenged right back with, according to what shiny-confident-robot? And no, first hand observation doesn't count, because that's mere anecdote.
  - elil17 4y ago
    It's an anecdote which is meant to illustrate a widely observed behavior among many humans: we're better at accounting for frequent errors than we are at accounting for rare ones. This is well established in the field of human reliability analysis (HRA). Take a look at the HRA method HEART (https://www.epd.gov.hk/eia/register/report/eiareport/eia_2242014/EIA/app/app12.10.pdf https://www.epd.gov.hk/eia/register/report/eiareport/eia_224...). There are three different error producing conditions (EPCs) related to rare failures (EPCs 1, 3, and 12 in table 1.3). As for my argument that people who frequently work with this technology aren't going to be so susceptible to "shiny-confident-robot" - that is based on my first hand observation. But the counterargument (people trust LLMs too much) is not based in data either. The Pearce, et. al paper everyone cites about security vulnerabilities didn't look at how much people trust the code, just how frequently it gives wrong answers (https://arxiv.org/pdf/2108.09293.pdf https://arxiv.org/pdf/2108.09293.pdf). An actual user study (n=20) showed that Copilot users engage critically with the tool by examining and testing suggestions, comparing multiple suggestions, and crosschecking it against documentation (https://arxiv.org/pdf/2206.15000.pdf https://arxiv.org/pdf/2206.15000.pdf).
- fakedang 4y ago
  They're not dumb. They're management consultants!
- jfk13 4y ago
  Experience suggests that people are all too willing to unthinkingly follow instructions just because "the computer said so". https://theweek.com/articles/464674/8-drivers-who-blindly-followed-gps-into-disaster https://theweek.com/articles/464674/8-drivers-who-blindly-fo...
  - ghaff 4y ago
    People have certainly done some dumb things in the GPS vein. That said, if you're taking confident navigation directions from either a person or a computer, it can be hard to make a split-second decision that something "doesn't seem right." (Obviously some of the more extreme cases still shouldn't happen.)
    - still_grokking 4y ago
      > People have certainly done some dumb things This sentence doesn't need to be continued. It's already the ultimate state of affairs since forever.
      - quicklime 4y ago
        That's true if your only goal is to generate truthful text. But if you're trying to generate useful text, continuing the sentence is necessary. Maybe that'll be the next step once LLMs get very good at generating truthful text :)
  - elil17 4y ago
    I would say GPS is in the zone of having such a low error rate that people can't process it right. If, as an example, GitHub Copilot added a security vulnerability in one in ten thousand lines of code, that would be a disaster because no one would double check it's work. Currently Copilot adds a security vulnerability in one in ten lines of code (made up number), so people with even a tad of experience using it know not to blindly trust it. I don't mean to dismiss the issue of over-trust in computer systems - I think it's a very serious issue, and one that needs to be addressed across a lot of disciplines.
bambax 4y ago
Exactly. It's quite bad. In the case of ChatGPT, it could probably be mitigated by prefacing each interaction or each answer with a warning "I'm just a machine that makes stuff up; don't take what I say too seriously". But for all the talk about "benefiting humanity" and being good to the world from so-called OpenAI, what we get instead is a constant affirmation that it's never wrong, with the upbeat tone of a dystopian policeman.