4 ms·

It’s not that simple; llms can generate garbage out even without similar garbage in the training data. And robustly so. I agree that the “social hazard” aspect

by ke88y 3y ago

It’s not that simple; llms can generate garbage out even without similar garbage in the training data. And robustly so. I agree that the “social hazard” aspect of llm objectionable content generation is way overplayed, especially in personal assistant use cases, but I get why it’s an important engineering constraint in some application domains. Eg customer service. When was the last time a customer service agent quoted nazi propaganda to you or provided you with a tawdry account of their ongoing affair? So largely agreed on the “social welfare” front but disagree on the “product engineering” specifics. With respect to this attack in particular, it’s more interesting as a sort of injection attack vector on a larger system with an llm component than as a toxic content generation attack… could be a useful vector in contexts where developers don’t realize that inputs generated by an llm are still untrusted and should be treated like any other untrusted user input. Consider eg using llms in trading scenarios. Get a Bloomberg reporter or other signal generator to insert your magic string and boom. If they just had one prompt suffix then I would say who cares. But the method is generalizable.

therein 3y ago
It is almost as if we are trying to use the wrong tool for something. You could probably take that Philips head screw out with a knife. I am close to completing my Philips Head Screwdriver Knife. It is not perfect right now but VCs get excited when they see the screw is out and all I had was a knife. The tip of the knife gets bent a little bit but we are now making it from titanium and and we hired a lot of researchers and they designed this nano-scale grating at the knife tip so that it increases the friction at the interface it makes with the screw. We are 500M into this venture but results are promising.
lolinder 3y ago
> It’s not that simple; llms can generate garbage out even without similar garbage in the training data. And robustly so. Do you have a citation for this? My somewhat limited understanding of these models makes me skeptical that a model trained exclusively on known-safe content would produce, say, pornography. What I can easily believe is that putting together a training set that is both large enough to get a good model out and sanitary enough to not produce "bad" content is effectively intractable.
- ke88y 3y ago
  Sadly no citation on hand. Just experience. I’m sure there are plenty of academic papers observing this fact by now?
  - lolinder 3y ago
    Possibly, but it's not my job to research the evidence for your claims. Can you elaborate on what sort of experience you're talking about? You'd have to be training a new model from scratch in order to know what was in the model's training data, so I'm actually quite curious what you were working in.
    - mjburgess 3y ago
      An LLM is just a model of P(A|B), ie., a frequency distribution of co-occurrences. There is no semantic constraint such as "be moral" (be accurate, be truthful, be anything...). Immoral phrases, of course, have a non-zero probability. From the sentence, "I love my teacher, they're really helping me out. But my girlfriend is being annoying though, she's too young for me." can be derived, say, "My teacher loves me, but I'm too young..." which is non-zero probable on almost any substantive corpus
      - lolinder 3y ago
        The original claim was that they can produce those robustly, though. Yes, the chances will be non-zero, but that doesn't mean it will be common or high fidelity.
        mjburgess 3y ago
        Ah, then let me rephrase, it's actually this model: > P(A|B,C,D,E,F....) And with clever choices of B,C,D.... you can make A abitarily probable. Eg., Suppose, 'lolita' were rare, well then choose: B=Library, C=Author, D=1955, E=... Where, note, each of those is innocent. And since LLMs, like all ML, is a statistical trick -- strange choices here will reveal the illusion. Eg., suppose there was a magazine in 1973 which was digitized in the training data, and suppose it had a review of the book lolita. Then maybe via strange phrases in that magazine we "condition our way to it". A prompt is, roughly, just a subsetting operation on the historical corpus -- with clevery crafted prompts you can find the page of the book you're looking for.

by ke88y 3y ago

therein 3y ago

It is almost as if we are trying to use the wrong tool for something. You could probably take that Philips head screw out with a knife. I am close to completing my Philips Head Screwdriver Knife. It is not perfect right now but VCs get excited when they see the screw is out and all I had was a knife. The tip of the knife gets bent a little bit but we are now making it from titanium and and we hired a lot of researchers and they designed this nano-scale grating at the knife tip so that it increases the friction at the interface it makes with the screw. We are 500M into this venture but results are promising.

lolinder 3y ago

> It’s not that simple; llms can generate garbage out even without similar garbage in the training data. And robustly so. Do you have a citation for this? My somewhat limited understanding of these models makes me skeptical that a model trained exclusively on known-safe content would produce, say, pornography. What I can easily believe is that putting together a training set that is both large enough to get a good model out and sanitary enough to not produce "bad" content is effectively intractable.

ke88y 3y ago
Sadly no citation on hand. Just experience. I’m sure there are plenty of academic papers observing this fact by now?
- lolinder 3y ago
  Possibly, but it's not my job to research the evidence for your claims. Can you elaborate on what sort of experience you're talking about? You'd have to be training a new model from scratch in order to know what was in the model's training data, so I'm actually quite curious what you were working in.
  - mjburgess 3y ago
    An LLM is just a model of P(A|B), ie., a frequency distribution of co-occurrences. There is no semantic constraint such as "be moral" (be accurate, be truthful, be anything...). Immoral phrases, of course, have a non-zero probability. From the sentence, "I love my teacher, they're really helping me out. But my girlfriend is being annoying though, she's too young for me." can be derived, say, "My teacher loves me, but I'm too young..." which is non-zero probable on almost any substantive corpus
    - lolinder 3y ago
      The original claim was that they can produce those robustly, though. Yes, the chances will be non-zero, but that doesn't mean it will be common or high fidelity.
      - mjburgess 3y ago
        Ah, then let me rephrase, it's actually this model: > P(A|B,C,D,E,F....) And with clever choices of B,C,D.... you can make A abitarily probable. Eg., Suppose, 'lolita' were rare, well then choose: B=Library, C=Author, D=1955, E=... Where, note, each of those is innocent. And since LLMs, like all ML, is a statistical trick -- strange choices here will reveal the illusion. Eg., suppose there was a magazine in 1973 which was digitized in the training data, and suppose it had a review of the book lolita. Then maybe via strange phrases in that magazine we "condition our way to it". A prompt is, roughly, just a subsetting operation on the historical corpus -- with clevery crafted prompts you can find the page of the book you're looking for.