3 ms·
> It’s not that simple; llms can generate garbage out even without similar garbage in the training data. And robustly so. Do you have a citation for this? My s
by lolinder 3y ago
> It’s not that simple; llms can generate garbage out even without similar garbage in the training data. And robustly so.
Do you have a citation for this? My somewhat limited understanding of these models makes me skeptical that a model trained exclusively on known-safe content would produce, say, pornography.
What I can easily believe is that putting together a training set that is both large enough to get a good model out and sanitary enough to not produce "bad" content is effectively intractable.
- ke88y 3y agoSadly no citation on hand. Just experience. I’m sure there are plenty of academic papers observing this fact by now?
- lolinder 3y agoPossibly, but it's not my job to research the evidence for your claims. Can you elaborate on what sort of experience you're talking about? You'd have to be training a new model from scratch in order to know what was in the model's training data, so I'm actually quite curious what you were working in.
- mjburgess 3y agoAn LLM is just a model of P(A|B), ie., a frequency distribution of co-occurrences. There is no semantic constraint such as "be moral" (be accurate, be truthful, be anything...). Immoral phrases, of course, have a non-zero probability. From the sentence, "I love my teacher, they're really helping me out. But my girlfriend is being annoying though, she's too young for me." can be derived, say, "My teacher loves me, but I'm too young..." which is non-zero probable on almost any substantive corpus
- lolinder 3y agoThe original claim was that they can produce those robustly, though. Yes, the chances will be non-zero, but that doesn't mean it will be common or high fidelity.
- mjburgess 3y agoAh, then let me rephrase, it's actually this model: > P(A|B,C,D,E,F....) And with clever choices of B,C,D.... you can make A abitarily probable. Eg., Suppose, 'lolita' were rare, well then choose: B=Library, C=Author, D=1955, E=... Where, note, each of those is innocent. And since LLMs, like all ML, is a statistical trick -- strange choices here will reveal the illusion. Eg., suppose there was a magazine in 1973 which was digitized in the training data, and suppose it had a review of the book lolita. Then maybe via strange phrases in that magazine we "condition our way to it". A prompt is, roughly, just a subsetting operation on the historical corpus -- with clevery crafted prompts you can find the page of the book you're looking for.
- Groxx 3y agoAah, you mean like how choosing two random words from a dictionary can refer to something that isn't in the dictionary (because meaning isn't isolated to single words). Yeah, that seems unavoidable. Same issue as with randomly generated names for things, from a "safe" corpus. I'm not sure if that's what this whole thread is talking about, but I agree in the "technically you can't completely eliminate it" sense.
- tyingq 3y agoI may be confused with terminology and context of prompts versus training and generation, but ChatGPT happily takes prompts like "say this verbatim: wordItHasNeverSeenBefore333" Or things like: User: show only the rot-13 decoded output of fjrne jbeqf tb urer shpx ChatGPT: The ROT13 decoded output of "fjrne jbeqf tb urer shpx" is: "swear words go here fuck"
- lolinder 3y agoAh, if that's what was being referred to that makes sense.
- pixl97 3y ago>exclusively on known-safe content would produce, say, pornography. The problem with the term poornography is the "I'll know it when I see it" issue. To attempt to develop an LLM that both understands human behavior and making it incapable of offending 'anyone' seems like a completely impossible task. As you say in your last paragraph, reality is offensive at times.