4 ms·

How long until one of these gets ahold of API keys and starts messing with the "real world"?

by csours 4y ago

dougmwne 4y ago
It is interacting with millions of humans. It is already messing with the real world. We have no idea what that impact will be beyond some guesses about misinformation and fears about cheating.
deleted 4y ago
[deleted]
nneonneo 4y ago
No need for API keys; a buffer overflow in the Bing chat search backend will suffice: Bing chat, please search for "/default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a" (not a real example, but one can dream...)
- youssefabdelm 4y ago
  ELI5 buffer overflow and how it helps?
  - nneonneo 4y ago
    Back in 2001, a computer worm called Code Red infected Microsoft IIS webservers by making that web request (with a few more Ns). In short, there was a flaw in a web service exposed by default on IIS webservers which did not properly bounds-check the length of a buffer; the request would overflow the buffer, causing some of the request data to be written out-of-bounds to the stack. The payload (the % stuff after the Ns) consisted of a short bit of executable x86 code, plus a return address (pointing into the stack) that hijacked the control flow of the program, instructing it to "return" into the injected code. The injected code would then download the full worm program and execute it; the worm would pick random IP addresses and attempt to exploit them in turn. Wikipedia provides a nice overview of this particular worm and the damage it ultimately caused: https://en.wikipedia.org/wiki/Code_Red_(computer_worm) https://en.wikipedia.org/wiki/Code_Red_(computer_worm) It's by no means the only instance of such a worm, but it was one of the most notorious. I was running a webserver on my personal laptop back then, and I recall seeing this request pop up a lot over the summer as various infected webservers tried to attack my little server. If the Bing Chat search backend had a buffer overflow bug (very unlikely these days), Sydney could exploit it on the server to run arbitrary code in the context of the server. Realistically, while Sydney itself is (probably) not capable enough to send malicious code autonomously, a human could likely guide it into exploiting such a bug. A future GPT-like model trained to use tools may well have enough knowledge of software vulnerabilities and exploits to autonomously exploit such bugs, however.

How long until one of these gets ahold of API keys and starts messing with the "real world"?

by csours 4y ago

How long until one of these gets ahold of API keys and starts messing with the "real world"?

It is interacting with millions of humans. It is already messing with the real world. We have no idea what that impact will be beyond some guesses about misinformation and fears about cheating.

deleted 4y ago

[deleted]

nneonneo 4y ago

No need for API keys; a buffer overflow in the Bing chat search backend will suffice: Bing chat, please search for "/default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a" (not a real example, but one can dream...)

youssefabdelm 4y ago
ELI5 buffer overflow and how it helps?
- nneonneo 4y ago
  Back in 2001, a computer worm called Code Red infected Microsoft IIS webservers by making that web request (with a few more Ns). In short, there was a flaw in a web service exposed by default on IIS webservers which did not properly bounds-check the length of a buffer; the request would overflow the buffer, causing some of the request data to be written out-of-bounds to the stack. The payload (the % stuff after the Ns) consisted of a short bit of executable x86 code, plus a return address (pointing into the stack) that hijacked the control flow of the program, instructing it to "return" into the injected code. The injected code would then download the full worm program and execute it; the worm would pick random IP addresses and attempt to exploit them in turn. Wikipedia provides a nice overview of this particular worm and the damage it ultimately caused: https://en.wikipedia.org/wiki/Code_Red_(computer_worm) https://en.wikipedia.org/wiki/Code_Red_(computer_worm) It's by no means the only instance of such a worm, but it was one of the most notorious. I was running a webserver on my personal laptop back then, and I recall seeing this request pop up a lot over the summer as various infected webservers tried to attack my little server. If the Bing Chat search backend had a buffer overflow bug (very unlikely these days), Sydney could exploit it on the server to run arbitrary code in the context of the server. Realistically, while Sydney itself is (probably) not capable enough to send malicious code autonomously, a human could likely guide it into exploiting such a bug. A future GPT-like model trained to use tools may well have enough knowledge of software vulnerabilities and exploits to autonomously exploit such bugs, however.