4 ms·

Right, but wouldn’t it only divulge information already available elsewhere (albeit less easily)?

by weakfish 4y ago

andrewflnr 4y ago
"less easily" matters a lot. That's the difference between one person finding the info and a thousand.
Sharlin 4y ago
Every single yet-undiscovered vulnerability in open-source software is "information already available elsewhere". (Closed-source as well, of course, but less easily reachable!) The bugs are there in the code, waiting to be found, they are not conjured out of thin air!
- weakfish 4y ago
  The issue is that chatgpt can’t reason together new information into a novel technical idea, only synthesize existing information (or hallucinate)
  - Sharlin 4y ago
    [Citation needed] Is it even possible to distinguish those two cases, or is it just shifting the goal posts / no true Scotsman fallacy? ("Okay, I admit that it can do X, but certainly it can't do Y which is definitely not the same as X because I say so") I'm pretty sure even GPT3/3.5/4 is perfectly able to spot many simple bugs and vulnerabilities in random code it's asked to review. Is there any reason to doubt that GPT(N+1) is able to do the same for much more subtle bugs in much larger codebases?
mdaniel 4y ago
https://greshake.github.io/ https://greshake.github.io/ (discussed https://news.ycombinator.com/item?id=34976886 https://news.ycombinator.com/item?id=34976886 ) summarized as > If allowed by the user, Bing Chat can see currently open websites. We show that an attacker can plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and exfiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser. which is/was also a prompt injection attack but one which had "real world" implications

4 ms·

Right, but wouldn’t it only divulge information already available elsewhere (albeit less easily)?

by weakfish 4y ago

Right, but wouldn’t it only divulge information already available elsewhere (albeit less easily)?

andrewflnr 4y ago
"less easily" matters a lot. That's the difference between one person finding the info and a thousand.
Sharlin 4y ago
Every single yet-undiscovered vulnerability in open-source software is "information already available elsewhere". (Closed-source as well, of course, but less easily reachable!) The bugs are there in the code, waiting to be found, they are not conjured out of thin air!
- weakfish 4y ago
  The issue is that chatgpt can’t reason together new information into a novel technical idea, only synthesize existing information (or hallucinate)
  - Sharlin 4y ago
    [Citation needed] Is it even possible to distinguish those two cases, or is it just shifting the goal posts / no true Scotsman fallacy? ("Okay, I admit that it can do X, but certainly it can't do Y which is definitely not the same as X because I say so") I'm pretty sure even GPT3/3.5/4 is perfectly able to spot many simple bugs and vulnerabilities in random code it's asked to review. Is there any reason to doubt that GPT(N+1) is able to do the same for much more subtle bugs in much larger codebases?
mdaniel 4y ago
https://greshake.github.io/ https://greshake.github.io/ (discussed https://news.ycombinator.com/item?id=34976886 https://news.ycombinator.com/item?id=34976886 ) summarized as > If allowed by the user, Bing Chat can see currently open websites. We show that an attacker can plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and exfiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser. which is/was also a prompt injection attack but one which had "real world" implications