3 ms·

> "We conducted extensive adversarial prompt injection testing, evaluating 123 test cases representing 29 different attack scenarios. " Doesn't this seem like

by asdff 1y ago

> "We conducted extensive adversarial prompt injection testing, evaluating 123 test cases representing 29 different attack scenarios. " Doesn't this seem like a remarkably small set of tests? And the fact that it took this testing to realize that prompt injection and giving the reigns to the AI agent is dangerous strikes me as strange that this wasn't anticipated while building the tool in the first place, before it even went to their red team. Move fast and break things I guess. Only it is the worlds largest browser and the risk of breaking things means financial ruin and/or the end of the internet as we know it as a human to human communication tool.

fwip 1y ago
And even after their mitigations on known attacks, the attacks were still successful 11% of the time! To misquote the IRA - "[Scammers] only need to be lucky once, you need to be lucky every time." Even a 1% chance of getting pwned every time you get sent a malicious email is way too high. Plus the scammers aren't gonna rest on their laurels - they'll be iterating too.
whatevertrevor 1y ago
I wonder how this will even fare in the review process, or if the big AI players will get a free pass here. My intuition says that it's a risk that Google/Chrome absolutely don't want to own, it will be curious to see how "Agentic" AI gets deployed in browsers from a liability fallout perspective.
- asdff 1y ago
  Probably no liability considering that is how other phishing attempts are viewed.
  - whatevertrevor 1y ago
    But in other phishing attempts the user actually gives out their password (unintentionally) to an unscrupulous actor. In this case there's a middle-man (the AI extension) doing that for you, sometimes without even confirming with you what you want. I think this is more akin to say a theoretical browser not implementing HTTPS properly so people's credentials/sessions can be stolen with MiTM attacks or something. Clearly the bad behavior is in the toolchain and not the user here, and I'm not sure how much you can wave away claiming "We told you it's not fully safe." You can't sell tomatoes that have a 10% chance of giving you food poisoning, even if you declare that chance on the label, you know?

by asdff 1y ago

fwip 1y ago

And even after their mitigations on known attacks, the attacks were still successful 11% of the time! To misquote the IRA - "[Scammers] only need to be lucky once, you need to be lucky every time." Even a 1% chance of getting pwned every time you get sent a malicious email is way too high. Plus the scammers aren't gonna rest on their laurels - they'll be iterating too.

whatevertrevor 1y ago

I wonder how this will even fare in the review process, or if the big AI players will get a free pass here. My intuition says that it's a risk that Google/Chrome absolutely don't want to own, it will be curious to see how "Agentic" AI gets deployed in browsers from a liability fallout perspective.

asdff 1y ago
Probably no liability considering that is how other phishing attempts are viewed.
- whatevertrevor 1y ago
  But in other phishing attempts the user actually gives out their password (unintentionally) to an unscrupulous actor. In this case there's a middle-man (the AI extension) doing that for you, sometimes without even confirming with you what you want. I think this is more akin to say a theoretical browser not implementing HTTPS properly so people's credentials/sessions can be stolen with MiTM attacks or something. Clearly the bad behavior is in the toolchain and not the user here, and I'm not sure how much you can wave away claiming "We told you it's not fully safe." You can't sell tomatoes that have a 10% chance of giving you food poisoning, even if you declare that chance on the label, you know?