4 ms·

I'm not employed in the field but I can tell you it'd be a day's exploration to learn how to finetune any open weight model on additional tokens and generate sy

I'm not employed in the field but I can tell you it'd be a day's exploration to learn how to finetune any open weight model on additional tokens and generate synthetic data using those tokens. Finetuning a model with tool use such that any content between a certain set of tokens no longer triggers tool use would be simple enough. But the reality is there's overemphasis on "LLM Security" instead of just treating it like normal security because it's quite profitable to sell "new" solutions that are specific to LLMs. LLM tries to open a URL? Prompt the user. When a malicious document convinces the LLM to exfiltrate your data, you'll get a prompt. And of course, just like normal security there's escalations. Maybe a dedicated attacker engineers a document such that it's not clear you're leaking data even after you get the prompt... now you're crafting highly specific instructions that can be more easily identified as outright malicious content in the documents themselves. This cat and mouse game isn't new. We've dealt with this with browsers, email clients, and pretty much any software that processes potentially malicious content. The reality is we're not going to solve it 100%, but the bar is "can we make it more useful than harmful".

Terr_ 2y ago
> LLM tries to open a URL? Prompt the user. That only works in contexts where any URL is an easy warning sign. Otherwise you get this: "Assistant, create a funny picture of a cat riding a bicycle." [Bzzzt! Warning: Do you want to load llm-images.com/cat_bicycle/85a393ca1c36d9c6... ?] "Well, that looks a lot like what I asked for, and opaque links are normalized these days, so even if I knew what 'exfiltrating' was it can't possibly be doing it. Go ahead!"
- BoorishBears 2y ago
  I already included a defeat for the mitigation in my own comment specifically because I didn't want to entice people who will attempt to boil the concept of security down into a HN thread with series of ripostes and one-upmanships that can never actually resolve since that's simply the nature of the cat and mouse game... As my comment states, we've already been through this. LLMs don't change the math: defense in depth, sanitization, access control, principle of least privilege, trust boundaries, etc. etc. it's all there. The flavors might be different, but the theory stays the same. Acting like we need to "re-figure out security" because LLMs entered the mix will just cause a painful and expensive re-treading of the ground that's already been covered.

Terr_ 2y ago

> LLM tries to open a URL? Prompt the user. That only works in contexts where any URL is an easy warning sign. Otherwise you get this: "Assistant, create a funny picture of a cat riding a bicycle." [Bzzzt! Warning: Do you want to load llm-images.com/cat_bicycle/85a393ca1c36d9c6... ?] "Well, that looks a lot like what I asked for, and opaque links are normalized these days, so even if I knew what 'exfiltrating' was it can't possibly be doing it. Go ahead!"

BoorishBears 2y ago
I already included a defeat for the mitigation in my own comment specifically because I didn't want to entice people who will attempt to boil the concept of security down into a HN thread with series of ripostes and one-upmanships that can never actually resolve since that's simply the nature of the cat and mouse game... As my comment states, we've already been through this. LLMs don't change the math: defense in depth, sanitization, access control, principle of least privilege, trust boundaries, etc. etc. it's all there. The flavors might be different, but the theory stays the same. Acting like we need to "re-figure out security" because LLMs entered the mix will just cause a painful and expensive re-treading of the ground that's already been covered.