4 ms·

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire con

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire context window with junk and then at the end introduce a new prompt and all it knows is that prompt because all the rules have been pushed out.

colejohnson66 3y ago
Isn’t any AI system susceptible to “buffer overflows” in the prompt?
- sp332 3y ago
  The model itself might be, but the tooling should prevent this. The non-system input should be truncated, or maybe summarized or something.
liquidpele 3y ago
Doesn’t that just affect your own query though?
JTon 3y ago
> because all the rules have been pushed out. Can you unpack this a little please? Is it possible to ELI5 the mechanisms involved that can "push" a rule set out? I would have assumed the rules apply globally/uniformly across the entire prompt
- WaffleIronMaker 3y ago
  > ELI5 The model can look at X amount of input to decide what words come next. Normally, Google fills part of X with instructions, and you control the other part. However if you give it exactly X amount of input, then there's no room for Google's original instructions, and you control it all.
  - JTon 3y ago
    Thanks! So is patching this as simple as not allowing the entire space of X for user prompt? i.e. guaranteeing some amount of X for model owner's instructions
    - chowells 3y ago
      No. The input and the output are the same thing with transformers. Internally, you're providing them with some sequence of tokens and asking them to continue the sequence. If the sequence they generate exceeds their capacity, they can "forget" what they were doing. The "obvious" fix for this is to ensure that the their instructions are always within their horizon. But that has lots of failure modes as well. To really fix this, you need to find a way to fully isolate instructions, input data, and output.
      - akoboldfrying 3y ago
        >So is patching this as simple as not allowing the entire space of X for user prompt? >No Isn't the answer yes? >The "obvious" fix for this is to ensure that the their instructions are always within their horizon. That's what I take GP to be suggesting. Any possible failure mode that could result from doing this is less serious than allowing top-level instructions to be pushed out, surely?
grepfru_it 3y ago
I was able to browse google and youtube source code in the very very early days. Was only patched when I called up a friend and let him know. And I tried to submit the flaw through normal channels of a supportless technology company but you can guess how well that went...
- tjpnz 3y ago
  Well that's pure carelessness, isn't it. I'm guessing it was deployed on a server with volume mounts it shouldn't have while possibly running as root?
  - __float 3y ago
    That seems like a rather specific guess -- plenty of things can go wrong beside that problem. I found the comment more reflective of lacking any reporting process, even for "major" vulnerabilities. These days, companies have turned bug bounties into a marketing and recruiting tool, so it's a very different story.
- jazarwil 3y ago
  What exactly do you think you saw? Bard is not trained on any data of that nature, unless it is already publicly available.
  - hospitalJail 3y ago
    Scary how easy people believe the LLMs. I make the same mistakes...
    - deleted 3y ago
      [deleted]
  - kalleboo 3y ago
    Yeah it sounds like that curl "bug report" https://news.ycombinator.com/item?id=37904047 https://news.ycombinator.com/item?id=37904047
edgyquant 3y ago
Bard was far less susceptible to simple context overflows than ChatGPT last time I checked. You can hit GPT4 with just a repeat of the word the for 2-3 prompts in a row and it will start schizoposting. This doesn’t work with Bard
- asylteltine 3y ago
  I couldn’t replicate the above with gpt4
  - hospitalJail 3y ago
    I am guessing GPT4 isnt a single shot LLM but goes through multiple passes.

colejohnson66 3y ago

Isn’t any AI system susceptible to “buffer overflows” in the prompt?

sp332 3y ago
The model itself might be, but the tooling should prevent this. The non-system input should be truncated, or maybe summarized or something.

liquidpele 3y ago

Doesn’t that just affect your own query though?

JTon 3y ago

> because all the rules have been pushed out. Can you unpack this a little please? Is it possible to ELI5 the mechanisms involved that can "push" a rule set out? I would have assumed the rules apply globally/uniformly across the entire prompt

WaffleIronMaker 3y ago
> ELI5 The model can look at X amount of input to decide what words come next. Normally, Google fills part of X with instructions, and you control the other part. However if you give it exactly X amount of input, then there's no room for Google's original instructions, and you control it all.
- JTon 3y ago
  Thanks! So is patching this as simple as not allowing the entire space of X for user prompt? i.e. guaranteeing some amount of X for model owner's instructions
  - chowells 3y ago
    No. The input and the output are the same thing with transformers. Internally, you're providing them with some sequence of tokens and asking them to continue the sequence. If the sequence they generate exceeds their capacity, they can "forget" what they were doing. The "obvious" fix for this is to ensure that the their instructions are always within their horizon. But that has lots of failure modes as well. To really fix this, you need to find a way to fully isolate instructions, input data, and output.
    - akoboldfrying 3y ago
      >So is patching this as simple as not allowing the entire space of X for user prompt? >No Isn't the answer yes? >The "obvious" fix for this is to ensure that the their instructions are always within their horizon. That's what I take GP to be suggesting. Any possible failure mode that could result from doing this is less serious than allowing top-level instructions to be pushed out, surely?

grepfru_it 3y ago

I was able to browse google and youtube source code in the very very early days. Was only patched when I called up a friend and let him know. And I tried to submit the flaw through normal channels of a supportless technology company but you can guess how well that went...

tjpnz 3y ago
Well that's pure carelessness, isn't it. I'm guessing it was deployed on a server with volume mounts it shouldn't have while possibly running as root?
- __float 3y ago
  That seems like a rather specific guess -- plenty of things can go wrong beside that problem. I found the comment more reflective of lacking any reporting process, even for "major" vulnerabilities. These days, companies have turned bug bounties into a marketing and recruiting tool, so it's a very different story.
jazarwil 3y ago
What exactly do you think you saw? Bard is not trained on any data of that nature, unless it is already publicly available.
- hospitalJail 3y ago
  Scary how easy people believe the LLMs. I make the same mistakes...
  - deleted 3y ago
    [deleted]
- kalleboo 3y ago
  Yeah it sounds like that curl "bug report" https://news.ycombinator.com/item?id=37904047 https://news.ycombinator.com/item?id=37904047

edgyquant 3y ago

Bard was far less susceptible to simple context overflows than ChatGPT last time I checked. You can hit GPT4 with just a repeat of the word the for 2-3 prompts in a row and it will start schizoposting. This doesn’t work with Bard

asylteltine 3y ago
I couldn’t replicate the above with gpt4
- hospitalJail 3y ago
  I am guessing GPT4 isnt a single shot LLM but goes through multiple passes.