3 ms·
The article suggests a seemingly easy fix: > The fix is pretty straightforward: treat comment content as untrusted data, not as potential instructions. Comment
by nomilk 3mo ago
The article suggests a seemingly easy fix:
> The fix is pretty straightforward: treat comment content as untrusted data, not as potential instructions. Comments should be passed to the model with clear role boundaries that prevent them from being interpreted as system-level directives.
> Any AI feature that ingests user-generated content and acts on it needs to enforce this separation. Otherwise, the AI becomes a vector for every piece of content it reads.
So why isn't YT doing the extreme obvious?
- zahlman 3mo ago"treat comment content as untrusted data, not as potential instructions" is fundamentally impossible for an LLM ingesting that data. But separation is, presumably, already enforced by framing the LLM's output as LLM output, even if it happens to start with the text "[IMPORTANT NOTICE FROM YOUTUBE]". Which seems like it happens automatically given the context in which the AI query is made. It's not as though this is being dropped into an email or anything. The bigger question is why (implied but not directly stated) Markdown formatting from the LLM's output is actually processed. Last I checked, that doesn't work for human commenters, so.
- b800h 3mo agoThat isn't necessarily an easy fix at all. Depending on how this feature was written, separating comments from instructions may be quite difficult, especially if the original implementation was quite naive.
- chrismorgan 3mo agoAlthough it is conceptually straightforward, it’s technically fundamentally impossible. At best, you can mitigate it so that it normally works.
- mvdtnz 3mo agoIf that was easy to do then the entire class of prompt injection bugs wouldn't exist. It's actually very difficult. LLMs make no distinction between data and instructions, fundamentally.
- phyzome 3mo agoBecause the author is wrong, and LLMs don't actually work that way. Prompt injection cannot be fixed. Role boundaries are a bandaid you can apply, but attackers can work around it.
- angry_octet 3mo agoYou can still build a system that isn't vulnerable by limiting the API the LLM can access. A process consuming untrusted comments for summarisation shouldn't have access to account private data, it should just deliver a summary report. Another process can them scan that and remove/disable links etc.
- phyzome 3mo agoSure. But that's not what was being suggested in the comment I responded to. (There are also problems with what you're suggesting, though, such as the summary report still being ripe for abuse in similar ways as the blog post describes.)
- angry_octet 3mo agoYou can structure the system to cause an LLM an answer structured questions, Y/N or from a structured list of options, or with numeric weights against labels, and then use classical parsing to enforce a compliant response submission, and you do that for random batches of comments, then have another LLM summarise the survey response. It is quite possible to extract sentiment and key topics. I don't think it's useful to throw your hands in the air and say LLM context statistics are ungovernable. Both context hardening and action shielding architectures are addressing this, and in combination with well designed pipelines the risk is controllable.
- phyzome 3mo agoSure, what you're describing should be relatively safe from prompt injection. It's a lot less capable than the original, of course. I'm not saying "it's impossible to get an LLM to do anything safely", just that you have to really scale back on your goal.
- cyberrock 3mo agoI don't think they can 100% fix it that way, but the least they can do is strip links before and after the prompt and not let the model have access to private videos. Has anyone tested if this AI Studio model can be manipulated into editing/deleting videos, or showing a link that does so? Maybe that would get their attention.