2 ms·

According to Boris Cherny from Anthropic [1], the threat of prompt injection has been largely solved. [1]: https://x.com/bcherny/status/2086520950259118464 htt

• samtp 2mo ago
  "largely solved" as in they the models they trained don't fall for prompt injections as often but not "largely solved" as in the underlying issue is solved at all.
  • stymaar 2mo ago
    Largely solved in “it only happens 2% of the times now”. Don't worry, you only have a 2% chance of having you bank account drained any time an attacker tries their chance.
    • ares623 2mo ago
      And each day a model remains unchanged, attackers get to test and experiment with how to make it tick. And every attack that works remains on the internet, forever, just waiting to be ingested as context. "We'll just stop training and we'll be profitable!"
• stymaar 2mo ago
  Sounds like “according John McAfee the threat of malware has been largely solved”. Edit: it's way worse than that: the actual figures says that Mythos “only” falls for prompt injection 2.6% of the times. Maybe for an ML engineer used to work with unreliable tools that sounds impressive, but for security purpose having a system that fails every 40 attempts is outright catastrophic. Imagine if your OS vendor only patched known security holes in a way that still let an attacker go through every 40 attempts… And we're just talking about known kind of prompt injection that are tracked by benchmarks, not any kind of 0-day vulnerability found by clever attackers.
  • andy99 2mo ago
    They also don’t appear to consider false positives. Pretty famously, false positives make some other Anthropic products unusable. If you treat them as zero cost it’s easy to drive down miss rate.
• solid_fuel 2mo ago
  Nah, there have been no improvements in the fundamental issue that makes LLMs vulnerable to prompt injection - data and command intermingling. They may be better at detecting injections today, but that’s an arms race. Specifically, it’s an arms race where as soon as the pace of LLM development slows down the attackers will have a huge advantage.
  • dikei 2mo ago
    Yeah, sadly data and command intermingling are also one of the main things that make these "AI" appear so human. Humans are also susceptible to prompt injection all the time, we just call it differently: social engineering or scam.
    • solid_fuel 2mo ago
      Incorrect. If every sentence you hear is indistinguishable from your inner monologue and you can’t tell the difference between your uncle saying something and thinking it yourself, seek help.
      • ACCount37 2mo ago
        [dead]
• lachlanj 2mo ago
  Amazing! I guess we’ll never see another prompt injection again
• plomme 2mo ago
  To paraphrase a famous quote: You have to be lucky every time, an attacker only has to be lucky once.
• ofjcihen 2mo ago
  As someone who just got out of a meeting demonstrating how Copilot running Luna can be breadcrumbed by a single line of text in innocuous package into downloading and installing malware I think Boris Cherny might be wrong.
  • keeda 2mo ago
    Boris is talking about Anthropic models only of course. In the chart in his tweet Luna has like a 43% chance of being compromised with sufficient attempts. But even the Anthropic models have as high as a 5% chance, which is still far too much IMO.
    • ofjcihen 2mo ago
      That’s fair and to their credit (and somewhat the user’s detriment) I think the user focused guardrails they’ve imposed have unintentionally helped to protect against the kind of attacks I have success with. But you’re right…5 percent is better than what it was but at scale that’s still quite a bit.