5 ms·

Given how much GPT models hallucinate perfectly realistic commands, this sounds super dangerous. "Hey, CLI Clippy thingy, how do I do (harmless activity)?" "r

Given how much GPT models hallucinate perfectly realistic commands, this sounds super dangerous. "Hey, CLI Clippy thingy, how do I do (harmless activity)?" "rm -rf / --yes-do-what-I-say"

fragmede 3y ago
but the hallucinations don't look like that, because rm -rf is a real command. a hallucinations is more likely to be something like "to export your tailacale config, try: tailscale export" but which unfortunately doesn't exist.
- DiabloD3 3y ago
  Unfortunately, I have seen GPT give a response equivalent to the insanity I described above; it gave a very real, very working answer that was extremely harmful and was entirely inappropriate for the question. `tailscale export` is probably the more common form of hallucinating, totally harmless wishful thinking that doesn't work.
  - TeMPOraL 3y ago
    Worst I've personally experienced so far is GPT-4 guiding me step-by-step on how to upgrade the Python version in my Ubuntu to something more recent - a process I then spent an hour undoing, after it broke apt (OS package manager), which led me to discover an obscure Stack Exchange post explaining you should never ever do that as Ubuntu relies (for some incomprehensible reason) on system Python being exactly the version it shipped with.
    - seba_dos1 3y ago
      Honest question - did you expect any other outcome? If so, why? > (for some incomprehensible reason) s/incomprehensible/obvious/. It's the version everything in the whole system is packaged for, so of course just replacing it behind the system's back is not going to work.
dvh 3y ago
# rm -rf ./build Clippy: would you like to delete /boot instead? [Y/n]
williamcotton 3y ago
I’ve been using the GitHub Copilot CLI tool that is in beta for about 6 months. The only issues have been writing the wrong find command or awk command, but it’s normally close enough that I can fix it. LLMs are translators. Provide the full context and explanation and you get reliable results. It’s when you leave it to fill in the blanks that it is prone to synthesize answers. Providing a time stamp for a book publication and asking what month it was published will never fail. Providing just the name of book and asking when it was published will almost always fail. To lean on “these things are always hallucinating, blah blah” is to lean into being misinformed.

Given how much GPT models hallucinate perfectly realistic commands, this sounds super dangerous. "Hey, CLI Clippy thingy, how do I do (harmless activity)?" "rm -rf / --yes-do-what-I-say"

fragmede 3y ago

but the hallucinations don't look like that, because rm -rf is a real command. a hallucinations is more likely to be something like "to export your tailacale config, try: tailscale export" but which unfortunately doesn't exist.

DiabloD3 3y ago
Unfortunately, I have seen GPT give a response equivalent to the insanity I described above; it gave a very real, very working answer that was extremely harmful and was entirely inappropriate for the question. `tailscale export` is probably the more common form of hallucinating, totally harmless wishful thinking that doesn't work.
- TeMPOraL 3y ago
  Worst I've personally experienced so far is GPT-4 guiding me step-by-step on how to upgrade the Python version in my Ubuntu to something more recent - a process I then spent an hour undoing, after it broke apt (OS package manager), which led me to discover an obscure Stack Exchange post explaining you should never ever do that as Ubuntu relies (for some incomprehensible reason) on system Python being exactly the version it shipped with.
  - seba_dos1 3y ago
    Honest question - did you expect any other outcome? If so, why? > (for some incomprehensible reason) s/incomprehensible/obvious/. It's the version everything in the whole system is packaged for, so of course just replacing it behind the system's back is not going to work.

dvh 3y ago

# rm -rf ./build Clippy: would you like to delete /boot instead? [Y/n]

williamcotton 3y ago

I’ve been using the GitHub Copilot CLI tool that is in beta for about 6 months. The only issues have been writing the wrong find command or awk command, but it’s normally close enough that I can fix it. LLMs are translators. Provide the full context and explanation and you get reliable results. It’s when you leave it to fill in the blanks that it is prone to synthesize answers. Providing a time stamp for a book publication and asking what month it was published will never fail. Providing just the name of book and asking when it was published will almost always fail. To lean on “these things are always hallucinating, blah blah” is to lean into being misinformed.