4 ms·

Frankly, you’re suffering from a serious failure of imagination if you think these things will just remain cute chatbots without any means of interacting with t

Frankly, you’re suffering from a serious failure of imagination if you think these things will just remain cute chatbots without any means of interacting with the outside world other than the user console. Indeed the cat’s already out of the bag with Bing. And you don’t even need that for the cute chatbot to be highly dangerous in the wrong hands. The first thing that trivially comes to mind is to convince GPT-(N+1) to find novel exploitable security vulnerabilities in OpenSSL or whatever. Strictly for responsible, white-hat purposes, of course. (In entirely unrelated news, a tool for loading entire code repos into GPT prompts currently ranks #2 on HN.)

quickthrower2 4y ago
The other thing could be the evolution of (or construction) of AI viruses: prompts that cause AI to send prompts to other AIs and so on.
zirgs 4y ago
Foreign intelligence agencies are going to use AI to find exploits. (If they aren't already). Forbidding our white hat hackers to defend our systems using AI makes no sense.
weakfish 4y ago
Right, but wouldn’t it only divulge information already available elsewhere (albeit less easily)?
- andrewflnr 4y ago
  "less easily" matters a lot. That's the difference between one person finding the info and a thousand.
- Sharlin 4y ago
  Every single yet-undiscovered vulnerability in open-source software is "information already available elsewhere". (Closed-source as well, of course, but less easily reachable!) The bugs are there in the code, waiting to be found, they are not conjured out of thin air!
  - weakfish 4y ago
    The issue is that chatgpt can’t reason together new information into a novel technical idea, only synthesize existing information (or hallucinate)
    - Sharlin 4y ago
      [Citation needed] Is it even possible to distinguish those two cases, or is it just shifting the goal posts / no true Scotsman fallacy? ("Okay, I admit that it can do X, but certainly it can't do Y which is definitely not the same as X because I say so") I'm pretty sure even GPT3/3.5/4 is perfectly able to spot many simple bugs and vulnerabilities in random code it's asked to review. Is there any reason to doubt that GPT(N+1) is able to do the same for much more subtle bugs in much larger codebases?
- mdaniel 4y ago
  https://greshake.github.io/ https://greshake.github.io/ (discussed https://news.ycombinator.com/item?id=34976886 https://news.ycombinator.com/item?id=34976886 ) summarized as > If allowed by the user, Bing Chat can see currently open websites. We show that an attacker can plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and exfiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser. which is/was also a prompt injection attack but one which had "real world" implications

quickthrower2 4y ago

The other thing could be the evolution of (or construction) of AI viruses: prompts that cause AI to send prompts to other AIs and so on.

zirgs 4y ago

Foreign intelligence agencies are going to use AI to find exploits. (If they aren't already). Forbidding our white hat hackers to defend our systems using AI makes no sense.

weakfish 4y ago

Right, but wouldn’t it only divulge information already available elsewhere (albeit less easily)?

andrewflnr 4y ago
"less easily" matters a lot. That's the difference between one person finding the info and a thousand.
Sharlin 4y ago
Every single yet-undiscovered vulnerability in open-source software is "information already available elsewhere". (Closed-source as well, of course, but less easily reachable!) The bugs are there in the code, waiting to be found, they are not conjured out of thin air!
- weakfish 4y ago
  The issue is that chatgpt can’t reason together new information into a novel technical idea, only synthesize existing information (or hallucinate)
  - Sharlin 4y ago
    [Citation needed] Is it even possible to distinguish those two cases, or is it just shifting the goal posts / no true Scotsman fallacy? ("Okay, I admit that it can do X, but certainly it can't do Y which is definitely not the same as X because I say so") I'm pretty sure even GPT3/3.5/4 is perfectly able to spot many simple bugs and vulnerabilities in random code it's asked to review. Is there any reason to doubt that GPT(N+1) is able to do the same for much more subtle bugs in much larger codebases?
mdaniel 4y ago
https://greshake.github.io/ https://greshake.github.io/ (discussed https://news.ycombinator.com/item?id=34976886 https://news.ycombinator.com/item?id=34976886 ) summarized as > If allowed by the user, Bing Chat can see currently open websites. We show that an attacker can plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and exfiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser. which is/was also a prompt injection attack but one which had "real world" implications