3 ms·

Google doesnt care about prompt injection attacks??? This is insane

by algoth1 3mo ago

tailscaler2026 3mo ago
They care. They'll fix it. They just won't pay the bounty for this bug.
- mapontosevenths 3mo ago
  I feel like it would be cheaper to pay a few bounties you dont really agree with than to risk a bad rep with security researchers.il Its still a relatively small community. Besides, if you don't pay the competition will, and ther use cases for your vulns are unlikely to be good for your business.
  - dylan604 3mo ago
    Google? And bad rep? Surely you jest
    - mapontosevenths 3mo ago
      I'm not and don't call me Shirley.
rwmj 3mo ago
Can they do anything about it? It's a fundamental flaw in how data is fed to LLMs. I'm getting PHP / SQL injection flashbacks.
- zahlman 3mo ago
  The described attack sounds like it's expecting the human to forget about having just clicked a UI element asking for a comment summary, and responding to a comment summary that tries to sound like an "important message from YouTube" as if it were actually such. It doesn't seem to involve the LLM actually having any agency to, for example, send an email to the creator. Mitigations would include ensuring it doesn't have that agency, and adding framing text to the reply, and perhaps disabling Markdown formatting of the reply. But also, the leak is being talked up quite a bit: > Private video titles aren't just metadata. They can reveal unreleased content, unannounced projects and sensitive personal material. Putting "sensitive personal material" in the title of a YouTube video upload and relying on YouTube to keep the video "private" seems like a terrible idea in the first place, and at best pointless.
  - Terr_ 3mo ago
    That sounds a bit like "nobody would ever fall for a phishing email." I don't think we should overestimate the technical sophistication and unceasing vigilance of the average YouTube user. Even if it's just a non-clickable link to "more information", some data can be exfiltrated that way.
    - zahlman 3mo ago
      > That sounds a bit like "nobody would ever fall for a phishing email." I don't think we should overestimate the technical sophistication and unceasing vigilance of the average YouTube user. By this standard, we shouldn't allow comments on YouTube. Or perhaps anywhere.
      - Terr_ 3mo ago
        That's equating regular social engineering versus LLM prompt injection and clicking a sneaky URL, I don't think those are equivalent scenarios or risks.
  - pa7ch 3mo ago
    Its not hard to imagine this is a serious risk in some cases. For example: A youtuber essentially working as a journalist made a big story recently about some illegal actions of a lying and litigious company (Bricks and Minifigs story). The youtuber has a 3rd video ready for when his gag order drops, if that were to be released early he could find himself in jail.
    - Paradigm2020 3mo ago
      Related to the bricks and minifigs story checkout the coffeezilla episode on it
- Terr_ 3mo ago
  Yep, and worse because the entire product relies on injection to operate, because everybody's excited about the "flexibility" of just telling it what your want.
- cobbal 3mo ago
  This is a case of lethal trifecta. This particular one can be fixed by either not giving the AI private data, or by removing the exfiltration opportunity. Why does the comment-summary bot need access to your private video ids? Why does it need to be able to output links? Most cases of prompt injection are harder to fix, and the success of the products they occur in relies on engineers who should know better sticking their heads in the sand about security risks.

Google doesnt care about prompt injection attacks??? This is insane

by algoth1 3mo ago

Google doesnt care about prompt injection attacks??? This is insane

tailscaler2026 3mo ago

They care. They'll fix it. They just won't pay the bounty for this bug.

mapontosevenths 3mo ago
I feel like it would be cheaper to pay a few bounties you dont really agree with than to risk a bad rep with security researchers.il Its still a relatively small community. Besides, if you don't pay the competition will, and ther use cases for your vulns are unlikely to be good for your business.
- dylan604 3mo ago
  Google? And bad rep? Surely you jest
  - mapontosevenths 3mo ago
    I'm not and don't call me Shirley.

rwmj 3mo ago

Can they do anything about it? It's a fundamental flaw in how data is fed to LLMs. I'm getting PHP / SQL injection flashbacks.

zahlman 3mo ago
The described attack sounds like it's expecting the human to forget about having just clicked a UI element asking for a comment summary, and responding to a comment summary that tries to sound like an "important message from YouTube" as if it were actually such. It doesn't seem to involve the LLM actually having any agency to, for example, send an email to the creator. Mitigations would include ensuring it doesn't have that agency, and adding framing text to the reply, and perhaps disabling Markdown formatting of the reply. But also, the leak is being talked up quite a bit: > Private video titles aren't just metadata. They can reveal unreleased content, unannounced projects and sensitive personal material. Putting "sensitive personal material" in the title of a YouTube video upload and relying on YouTube to keep the video "private" seems like a terrible idea in the first place, and at best pointless.
- Terr_ 3mo ago
  That sounds a bit like "nobody would ever fall for a phishing email." I don't think we should overestimate the technical sophistication and unceasing vigilance of the average YouTube user. Even if it's just a non-clickable link to "more information", some data can be exfiltrated that way.
  - zahlman 3mo ago
    > That sounds a bit like "nobody would ever fall for a phishing email." I don't think we should overestimate the technical sophistication and unceasing vigilance of the average YouTube user. By this standard, we shouldn't allow comments on YouTube. Or perhaps anywhere.
    - Terr_ 3mo ago
      That's equating regular social engineering versus LLM prompt injection and clicking a sneaky URL, I don't think those are equivalent scenarios or risks.
- pa7ch 3mo ago
  Its not hard to imagine this is a serious risk in some cases. For example: A youtuber essentially working as a journalist made a big story recently about some illegal actions of a lying and litigious company (Bricks and Minifigs story). The youtuber has a 3rd video ready for when his gag order drops, if that were to be released early he could find himself in jail.
  - Paradigm2020 3mo ago
    Related to the bricks and minifigs story checkout the coffeezilla episode on it
Terr_ 3mo ago
Yep, and worse because the entire product relies on injection to operate, because everybody's excited about the "flexibility" of just telling it what your want.
cobbal 3mo ago
This is a case of lethal trifecta. This particular one can be fixed by either not giving the AI private data, or by removing the exfiltration opportunity. Why does the comment-summary bot need access to your private video ids? Why does it need to be able to output links? Most cases of prompt injection are harder to fix, and the success of the products they occur in relies on engineers who should know better sticking their heads in the sand about security risks.