4 ms·

> Attacker leaves the comment on a creator's video. > Creator opens YouTube studio's comment tab. > Creator clicks a suggested AI prompt (Designed by YouTube)

by wxw 3mo ago

> Attacker leaves the comment on a creator's video. > Creator opens YouTube studio's comment tab. > Creator clicks a suggested AI prompt (Designed by YouTube) > Injection fires, attacker-controlled content appears in the response. It's insane that YouTube doesn't see prompt injection as a bug.

Dylan16807 3mo ago
Yeah, if going to site and just clicking a link given to me by the site itself is getting socially engineered, then something is very wrong with that site.
- krackers 3mo ago
  Youtube comments are also links given by the site. I think in this case it's not necessarily the prompt injection that's the issue but the fact that untrusted content allows formatted links. YouTube doesn't allow clicabkle links in comments iirc, so the same needs to be applied here.
  - Dylan16807 3mo ago
    If comments allowed links in general, this would be one step less egregious, but it would still be a huge issue if clicking a comment link could leak private information. The fact that the prompt injection can customize the link before giving it to the user is the bulk of the problem here. If it just regurgitated a link it would be a flaw but a notably smaller flaw.
  - jdiff 3mo ago
    Those are pretty clearly delineated as user-generated content, and also aren't able to be modified to include information that the malicious user doesn't have another way of accessing.
muldvarp 3mo ago
Well prompt injection is pretty much unfixable. So if they actually saw this as a security vulnerability they would have to remove this feature.
- afarah1 3mo ago
  Couple of things that could be done, from the top of my head: - Strip links, script tags, etc - Apply the same filters used in user comments - Add a warning indicating user-generated content may be present The post suggests the UX is problematic in that it allows user-generated links to pass as YouTube generated content. I'm not familiar with Creator Studio to know if this is the case, but if so, simple changes can go a long way.
  - ireadmevs 3mo ago
    I would even go one step further: why would an AI tasked with summarizing external comments require any tool access, especially being able to reach into non public data? The solution to prompt injection is not more AI on top of it, it starts with data access controls
jdiff 3mo ago
It opens a can of worms for them if they do consider prompt injection a bug because there's ultimately no defense. If they accept this, there are instantly hundreds of other moles they now have to whack or pay out for. Or dismiss them all as social engineering and keep it moving.
- orbital-decay 3mo ago
  >because there's ultimately no defense Kind of? It's not fixable as a spherical class of attacks in vacuum, but you can do a lot to mitigate particular cases, and in most cases you can patch unnecessary side channels for the injection to reach the context in an unintended way.
  - keepamovin 3mo ago
    Isn’t it trivially fixable by having a monitor LLM? The monitor just reviews each turn pair and asks, “Is this conversation being manipulated via prompt injection?”
    - zapkyeskrill 3mo ago
      Is it? Or does it just make it multi dimensional? As in, prompt now need to anticipate there being a monitor and instruct that one too, indirectly.
      - keepamovin 3mo ago
        Right - but that sounds too intractable to hold up. See my other comment, I feel a chain of monitors defeats it. But hey! Who knows?
        jdiff 3mo ago
        An n-deep chain of monitors doesn't really have any defense that an (n-1)-deep chain of monitors has. None of them have the capacity to separate data and instructions. All you're doing is (in some ways) giving the model more rolls of the dice to catch what's going on, but the kind of dice and the needed values to roll are in the attacker's hands as much as yours.
latexr 3mo ago
> It's insane that YouTube doesn't see prompt injection as a bug. Insane but not unexpected, from the company who literally sang at us that “there’s no wrong way to prompt”. https://www.youtube.com/watch?v=9bBfYX8X5aU&t=48s https://www.youtube.com/watch?v=9bBfYX8X5aU&t=48s
IshKebab 3mo ago
I dunno this seems like a quite far fetched attack with minimal impact in the very unlikely case that it succeeds.
Ozzie_osman 3mo ago
An org that big doesn't "see". A triager with very little context and authority is probably the one making this call and my guess is the process is failing to direct or escalate it to the right person.

by wxw 3mo ago

Dylan16807 3mo ago

Yeah, if going to site and just clicking a link given to me by the site itself is getting socially engineered, then something is very wrong with that site.

krackers 3mo ago
Youtube comments are also links given by the site. I think in this case it's not necessarily the prompt injection that's the issue but the fact that untrusted content allows formatted links. YouTube doesn't allow clicabkle links in comments iirc, so the same needs to be applied here.
- Dylan16807 3mo ago
  If comments allowed links in general, this would be one step less egregious, but it would still be a huge issue if clicking a comment link could leak private information. The fact that the prompt injection can customize the link before giving it to the user is the bulk of the problem here. If it just regurgitated a link it would be a flaw but a notably smaller flaw.
- jdiff 3mo ago
  Those are pretty clearly delineated as user-generated content, and also aren't able to be modified to include information that the malicious user doesn't have another way of accessing.

muldvarp 3mo ago

Well prompt injection is pretty much unfixable. So if they actually saw this as a security vulnerability they would have to remove this feature.

afarah1 3mo ago
Couple of things that could be done, from the top of my head: - Strip links, script tags, etc - Apply the same filters used in user comments - Add a warning indicating user-generated content may be present The post suggests the UX is problematic in that it allows user-generated links to pass as YouTube generated content. I'm not familiar with Creator Studio to know if this is the case, but if so, simple changes can go a long way.
- ireadmevs 3mo ago
  I would even go one step further: why would an AI tasked with summarizing external comments require any tool access, especially being able to reach into non public data? The solution to prompt injection is not more AI on top of it, it starts with data access controls

jdiff 3mo ago

It opens a can of worms for them if they do consider prompt injection a bug because there's ultimately no defense. If they accept this, there are instantly hundreds of other moles they now have to whack or pay out for. Or dismiss them all as social engineering and keep it moving.

orbital-decay 3mo ago
>because there's ultimately no defense Kind of? It's not fixable as a spherical class of attacks in vacuum, but you can do a lot to mitigate particular cases, and in most cases you can patch unnecessary side channels for the injection to reach the context in an unintended way.
- keepamovin 3mo ago
  Isn’t it trivially fixable by having a monitor LLM? The monitor just reviews each turn pair and asks, “Is this conversation being manipulated via prompt injection?”
  - zapkyeskrill 3mo ago
    Is it? Or does it just make it multi dimensional? As in, prompt now need to anticipate there being a monitor and instruct that one too, indirectly.
    - keepamovin 3mo ago
      Right - but that sounds too intractable to hold up. See my other comment, I feel a chain of monitors defeats it. But hey! Who knows?
      - jdiff 3mo ago
        An n-deep chain of monitors doesn't really have any defense that an (n-1)-deep chain of monitors has. None of them have the capacity to separate data and instructions. All you're doing is (in some ways) giving the model more rolls of the dice to catch what's going on, but the kind of dice and the needed values to roll are in the attacker's hands as much as yours.

latexr 3mo ago

> It's insane that YouTube doesn't see prompt injection as a bug. Insane but not unexpected, from the company who literally sang at us that “there’s no wrong way to prompt”. https://www.youtube.com/watch?v=9bBfYX8X5aU&t=48s https://www.youtube.com/watch?v=9bBfYX8X5aU&t=48s

IshKebab 3mo ago

I dunno this seems like a quite far fetched attack with minimal impact in the very unlikely case that it succeeds.

Ozzie_osman 3mo ago

An org that big doesn't "see". A triager with very little context and authority is probably the one making this call and my guess is the process is failing to direct or escalate it to the right person.