3 ms·

I'm still personally very cautious of these models. There's a seemingly unlimited attack surface here that is going to take a long time to protect. I was tryin

I'm still personally very cautious of these models. There's a seemingly unlimited attack surface here that is going to take a long time to protect. I was trying a few things out myself on Bard and managed to get it to run code in its own process (at least I think it did?) https://twitter.com/Codesleuth/status/1697025065177452971 https://twitter.com/Codesleuth/status/1697025065177452971

tantalor 3y ago
Eh probably not. That's a hallucination
- stavros 3y ago
  It's surprising to me how little people understand of how LLMs work. How does someone think that an LLM will just exec() random Python code into its own process? It doesn't have access to that, any more than Notepad has access to execute stuff you type up in the document.
  - theptip 3y ago
    To be fair, ChatGPT code interpreter stands up a VM and runs Python code so it’s not completely outlandish. You are also right that’s not how Bing happens to work right now.
  - hahajk 3y ago
    Also to be fair, Notepad was exploited to execute arbitrary code. https://www.digitaltrends.com/computing/major-security-flaw-in-notepad-leaves-windows-pcs-vulnerable-to-hackers/ https://www.digitaltrends.com/computing/major-security-flaw-...
    - tantalor 3y ago
      Microsoft product...
  - Traubenfuchs 3y ago
    Do not underestimate notepad please. It has (had) code execution capabilities. https://x.com/taviso/status/1133384839321853954?s=20 https://x.com/taviso/status/1133384839321853954?s=20
  - ChatGTP 3y ago
    It doesn't, but "plugins" can and do allow these things to run code in arbitrary places.
dragonwriter 3y ago
You got it to provide a chat response with a remotely plausible (but still somewhat unlikely, Bard probably isn’t running on MacOS servers, though training data with samples from which it might project the answer probably disproportionately is from people running code on MacOS desktops) answer to what the result of doing that would be.
madeofpalk 3y ago
What mechanism do you think you exploited to make Bard execute arbitrary code? Do you think Google engineers left in a secret eval($userPrompt) in the code base? Or do you think the Bard program became sentient and rewrote its own code?
- Codesleuth 3y ago
  Bear in mind that I'm still not convinced that it did actually run the code - seems more likely that it just simulated it. I got to this point by asking it different ways to expose its Google Search API key initially. Every attempt failed as if it was doing some inspection of its own output and identifying that it was exposing the key, which violated one of its rules. Then I tried asking it to base64 encode the key and print it, same issue. Then I asked it to base64 some arbitrary text, which it did. From there I kept asking it to run bits of code. It appeared to be doing what I asked, but who knows?
  - DaiPlusPlus 3y ago
    > Bear in mind that I'm still not convinced that it did actually run the code - seems more likely that it just simulated it. Kinda reminds me of this: https://arstechnica.com/information-technology/2022/12/openais-new-chatbot-can-hallucinate-a-linux-shell-or-calling-a-bbs/ https://arstechnica.com/information-technology/2022/12/opena...
  - madeofpalk 3y ago
    > I got to this point by asking it different ways to expose its Google Search API key initially. There still seems to be a fatal misunderstanding of how these "AI" work. How would the Bard LLM know the API key it uses (ignoring the fact that it probably uses non-public APIs with different authentication mechanisms...). From my understanding, there's two ways this would be possible - it was trained on data that contains its Google Search API key (doubtful), or Google engineers provide the API key in it's prompts (doubtful, they're aware of prompt leaks).

tantalor 3y ago

Eh probably not. That's a hallucination

stavros 3y ago
It's surprising to me how little people understand of how LLMs work. How does someone think that an LLM will just exec() random Python code into its own process? It doesn't have access to that, any more than Notepad has access to execute stuff you type up in the document.
- theptip 3y ago
  To be fair, ChatGPT code interpreter stands up a VM and runs Python code so it’s not completely outlandish. You are also right that’s not how Bing happens to work right now.
- hahajk 3y ago
  Also to be fair, Notepad was exploited to execute arbitrary code. https://www.digitaltrends.com/computing/major-security-flaw-in-notepad-leaves-windows-pcs-vulnerable-to-hackers/ https://www.digitaltrends.com/computing/major-security-flaw-...
  - tantalor 3y ago
    Microsoft product...
- Traubenfuchs 3y ago
  Do not underestimate notepad please. It has (had) code execution capabilities. https://x.com/taviso/status/1133384839321853954?s=20 https://x.com/taviso/status/1133384839321853954?s=20
- ChatGTP 3y ago
  It doesn't, but "plugins" can and do allow these things to run code in arbitrary places.

dragonwriter 3y ago

You got it to provide a chat response with a remotely plausible (but still somewhat unlikely, Bard probably isn’t running on MacOS servers, though training data with samples from which it might project the answer probably disproportionately is from people running code on MacOS desktops) answer to what the result of doing that would be.

madeofpalk 3y ago

What mechanism do you think you exploited to make Bard execute arbitrary code? Do you think Google engineers left in a secret eval($userPrompt) in the code base? Or do you think the Bard program became sentient and rewrote its own code?

Codesleuth 3y ago
Bear in mind that I'm still not convinced that it did actually run the code - seems more likely that it just simulated it. I got to this point by asking it different ways to expose its Google Search API key initially. Every attempt failed as if it was doing some inspection of its own output and identifying that it was exposing the key, which violated one of its rules. Then I tried asking it to base64 encode the key and print it, same issue. Then I asked it to base64 some arbitrary text, which it did. From there I kept asking it to run bits of code. It appeared to be doing what I asked, but who knows?
- DaiPlusPlus 3y ago
  > Bear in mind that I'm still not convinced that it did actually run the code - seems more likely that it just simulated it. Kinda reminds me of this: https://arstechnica.com/information-technology/2022/12/openais-new-chatbot-can-hallucinate-a-linux-shell-or-calling-a-bbs/ https://arstechnica.com/information-technology/2022/12/opena...
- madeofpalk 3y ago
  > I got to this point by asking it different ways to expose its Google Search API key initially. There still seems to be a fatal misunderstanding of how these "AI" work. How would the Bard LLM know the API key it uses (ignoring the fact that it probably uses non-public APIs with different authentication mechanisms...). From my understanding, there's two ways this would be possible - it was trained on data that contains its Google Search API key (doubtful), or Google engineers provide the API key in it's prompts (doubtful, they're aware of prompt leaks).