4 ms·

Can't the attacker then jailbreak the first LLM to generate jailbreak with actions for the second one?

• arthurcolle 1y ago
  Yes they can
  • ares623 1y ago
    Hmm so we need 3 LLMs
    • zwnow 1y ago
      Doesn't help. https://gandalf.lakera.ai/baseline https://gandalf.lakera.ai/baseline This thing models exactly these scenarios and asks you to break it, its still pretty easy. LLMs are not safe.
• dfabulich 1y ago
  If you read the fine article, you'll see that the approach includes a non-LLM controller managing structured communication between the Privileged LLM (allowed to perform actions) and the Quarantined LLM (only allowed to produce structured data, which is assumed to be tainted). See also CaMeL https://simonwillison.net/2025/Apr/11/camel/ https://simonwillison.net/2025/Apr/11/camel/ which incorporates a type system to track tainted data from the Quarantined LLM, ensuring that the Privileged LLM can't even see tainted _data_ until it's been reviewed by a human user. (But this can induce user fatigue as the user is forced to manually approve all the data that the Privileged LLM can access.)
  • yencabulator 1y ago
    "Structured data" is kind of the wrong description for what Simon proposes. JSON is structured but can smuggle a string with the attack inside it. Simon's proposal is smarter than that.
• j45 1y ago
  One would have to be relatively invisible. Non-deterministic security feels like a relatively new area.