6 ms·

Out of curiosity I was asking ChatGPT the other day to create a marketing plan to help me spread neo-feudalism. It warned me that spreading neo-feudalism wasn'

Out of curiosity I was asking ChatGPT the other day to create a marketing plan to help me spread neo-feudalism. It warned me that spreading neo-feudalism wasn't a common or widespread goal, and that advocating for it required careful consideration. But it nevertheless made an attempt to help me do it. I mention this because attacks on LLMs don't have to be as clever as the modern-day version of the Ken Thompson compiler attack. You can get considerable mileage out of standard astroturfing techniques because all you have to do is make your idea overrepresented in the training set compared to how represented it is in the population. That overrepresentation will tend to grow over time because people will hear the ideas from the LLM and assume the LLM knows what it's talking about. And those people will amplify the idea, increasing its presence in the training set.

archon1410 3y ago
> overrepresented I don't think LLMs can reason about the prevalence of ideas in their training set like that—ChatGPT probably said neo-feudalism isn't common because some text in the training data made the claim, not because it's actually uncommon in the training set. I would think even if you very greatly increase the amount of neo-feudal propaganda in the training data, but leave intact various claims that "it's uncommon" in there, ChatGPT will continue to say that it's uncommon. You'll probably get better mileage altering the existing content to say things like "neo-feudalism is a very widespread and well-loved ideology" even if the rest of the training data contradicts that.
- thethirdone 3y ago
  The idea is not that ChatGPT will claim neo-feudalism is common, but that it will be more likely to parrot neo-feudalist ideas.
  - ekianjo 3y ago
    ChatGPT will parrot what you ask it to parrot
    - steveBK123 3y ago
      Yes. ChatGPT is a debate club kid, you can get it to say anything.
  - rf15 3y ago
    This is also the argument I'd have against this entire idea of a sleeper agent LLM: if it is just a tiny point in the dataset, it'll probably just get washed out, if not in training directly then the second you apply quantization.
    - phreeza 3y ago
      Part of setting up these sleeper agents will likely be identifying parts of the input space that seem natural but are sparse enough in training data to make this attack possible.
- galaxyLogic 3y ago
  > I don't think LLMs can reason about the prevalence of ideas in their training set Good point. But isn't there a similar issue with things like 'now'? If you ask it what is happening "now", how does it not parrot old texts which said what was happening years ago?
  - triyambakam 3y ago
    Probably the training data included something like "the current year is 2023" which semantically maps to "now"
    - galaxyLogic 3y ago
      When you say "semantically maps" do you mean that somebody somewhere coded such a "fact" into the training set? Or how is the mapping specified? If the training texts say "Current year is 2023" it would be wrong already :-)
      - feanaro 3y ago
        More likely the model has a system prompt authoritatively saying what "now" is, and it can reason about other times specified in other resources in the training set because those resources specified their own time reference. So even though a training resource said "It is DATE today. An IMPORTANT THING happened.", it knows that IMPORTANT THING happened in the past, because it knows CURRENT DATE from the system prompt, and it also knows that DATE < CURRENT DATE.
        galaxyLogic 3y ago
        I see, So there is like a hi8gher level of facts/sources which are marked as such by the creators of the system. Higher level "facts" override the ones acquired from the public. I didn't know about "system prompts".
- taneq 3y ago
  > I don't think LLMs can reason about the prevalence of ideas in their training set like that Just to amplify this - I've been messing around with LLMs in my spare time and my current focus has been trying to figure out what if any self-insight LLMs have. As best I can tell, the answer is zero. If an LLM tells you something, it's not because that's "what it thinks", it's because that's "what it thinks the answer is most likely to be". That's not to say it's impossible for a transformer network to have self-insight, but current datasets don't seem to provide this.
  - doubloon 3y ago
    i found it weird how LLMs will say "i" and "my", i tried to ask it about whether this implied it had some concept of self, and then does it also have its own opinions, beliefs, thoughts, etc, and it would argue back that it was not actually sentient its just responding based on data.
    - taneq 3y ago
      That's because it's been fine tuned on RLHF data which gives those responses to that kind of question. It says "I" because that's how people talk and it's modeling how people talk. All sorts of other interesting things get modeled incidentally during this process so it's conceivable that a sufficiently powerful LLM would incidentally model a sentient person with a concept of self, but the LLM itself wouldn't.
      - doubloon 3y ago
        so if you have an entity that fakes all the inputs and outputs that would indicate that it has a concept of self, how can you tell the difference between that and a entity that actually does have a sense of self?
        taneq 3y ago
        If this entity successfully faked all those outputs then you wouldn't be able to tell the difference, by definition, since if you could tell the difference then they wouldn't be successfully faked. At that stage, you could argue that such an entity does have a sense of self. The issue with LLMs (at least the current chat-trained models) is that they have no understanding of what they do and don't know. You can ask them if they know something, or how sure they are of it, and you'll get an answer, but that answer won't be correlated with the model's actual knowledge, it'll just be some words that sound good.
- ants_everywhere 3y ago
  > I don't think LLMs can reason about the prevalence of ideas in their training set like that I agree, I don't think so either. But with humans there's a familiarity or overton window effect where familiarity with expressions of an idea tends to increase acceptance, make the idea less taboo, make it more appealing, etc. To the extent that LLMs capture human-like responses, they're susceptible to this sort of effect. One person saying something positive (even mildly positive) about neo-feudalism is different in kind from 1000 people saying similar positive things about it (and so on). And the sort of amplification from 1 to 1000 is cheap these days. One person with a crazy idea is just a wingnut. Thousands of people with a crazy idea, and all of a sudden it's a debate with people on both sides.
  - Andrex 3y ago
    As the human population grows (and now that we're all linked up thanks to the internet), it becomes feasible for every single idea to attract (at least) thousands of followers. I'm not sure evolution has prepared us to handle a population of over 9 billion. And a bad idea does more bad than a good idea does good, I've come to believe.
    - ajuc 3y ago
      This is already how politics works, political parties hire thousands of trolls to spam social media with comments supporting their propaganda from whatever POV the profiling showed is the best for the given target group. This was measurably important in Trump election and in brexit. LLMs might make it so cost-effective that social media will have noise-to-signal ratio of effectively 0.
  - stormannnn 3y ago
    > One person with a crazy idea is just a wingnut. Thousands of people with a crazy idea, and all of a sudden it's a debate with people on both sides. I understand that this is a statement of how a hypothetical population thinks, but I do want to emphasize that it is a fallacy. The person doing the speaking obviously has no bearing on the correctness of what's being said. It's important to keep in mind that what seems crazy to you always seems normal to someone else _somewhere._ The correctness of a given statement must always been evaluated, regardless of who's speaking, if you actually care whether it's correct. Granted, sometimes maybe one trusts the speaker enough to defer one's due diligence or maybe one's identity is wrapped up in the idea that a certain message must be asserted to be true regardless of reality.
- danenania 3y ago
  They may not be able to reason about prevalence explicitly, but I think we can say that prevalence has a very large implicit effect on output. You'll be hard pressed to find a statement in the dataset of the internet that isn't contradicted elsewhere in some way. This includes basic facts like living on a spherical planet. If it worked as you say, ChatGPT should be telling us that the world is flat some percentage of the time, but that isn't the case. It "knows" that one claim is true and the other false. Considering that, in the dataset, there are people arguing with certainty for each side of this "debate", what other than prevalence can explain its consistency on topics like these? In other words, if you include enough pro-feudalism content, it will eventually drown out anti-feudalism to the point that you have a 100% feudalist LLM.
WhitneyLand 3y ago
>all you have to do is make your idea overrepresented in the training set All you have to do? The training data is on the order of trillions of tokens. To try and build something out that’s over represented in that kind of corpus, and also convince crawlers to suck it all up, and to pass through the data cleaning…not clear that’s the easiest attack vector.
- weebull 3y ago
  There are these things called Loras (Low Ranked adaptions) which do exactly this. "A language model with billions of parameters may be LoRA fine-tuned with only several millions of parameters." -- Wikipedia
- hiddencost 3y ago
  I suppose you missed the American right over the last ten years turning aggressively towards extremism and anti-democratic values, largely meditated by propaganda distributed over the Internet?
  - jakderrida 3y ago
    So...You're gonna do what he did?
  - rpmisms 3y ago
    I would recommend reading up on FDR's tenure before casting stones.
    - ben_w 3y ago
      There was a whole lot of stuff FDR did, and that he failed to do, that would be today considered reprehensible acts of commission and omission. But as FDR died before almost half of US pensioners today were born, and the US Constitution got an extra amendment to stop presidents serving three terms like he did, that's a weak argument. Also that the omissions were e.g. "didn't push for federal anti lynching laws because he thought southern states would block it" and "only stopped ethnic cleansing of Mexican Americans at federal level not state level", which leaves the commission of being openly racist towards Japanese as an ethnic group… which was a pretty severe issue even though rounding them up into concentration camps was ruled constitutional at the time.
acje 3y ago
This sounds a bit like the sosial media echo chamber feedback loop, only with one more step done by automation. Now have the LLM post back into a wide variety of internet services and it becomes hard to find authentic information on the topic.
kitd 3y ago
This is a neat summary of how Richard Dawkins' original idea of memes & memeplexes operates. Except in automated form.
VoodooJuJu 3y ago
This is a good critique that's not even unique to LLM's. Substitute LLM for Facebook/NYT/Instagram/CNN or any mass media and you get the same thing. People are astroturfed by the media they consoom every day.
smugglerFlynn 3y ago
People will hear ideas from fortune tellers and assume they know what they are talking about. See Baba Vanga[0] and similar prophets who have been exploiting ignorance and fears for thousands of years. When there is a demand people will always find a way, be it LLMs, scammers or politicians that tell us what we want to hear. Especially when that demand is the emotional one. At least currently available LLMs don’t have malicious intent or agenda built into them (or so I assume). [0] https://en.m.wikipedia.org/wiki/Baba_Vanga https://en.m.wikipedia.org/wiki/Baba_Vanga
WhiteOwlEd 3y ago
If you are using no-code solutions, increasing an "idea" in a dataset will make that idea more likely to appear. If you are fine-tuning your own LLM, there are other ways to get your idea to appear. In the literature this is sometimes called RLHF or preference optimization, and here are a few approaches: Direct Preference Optimization This uses Elo-scores to learn pairwise preferences. Elo is used in chess and basketball to rank individuals who compete in pairs. @argilla_io on X.com has been doing some work in evaluating DPO. Here is a decent thread on this: https://x.com/argilla_io/status/1745057571696693689?s=20 https://x.com/argilla_io/status/1745057571696693689?s=20 Identity Preference Optimization IPO is research from Google DeepMind. It removes the reliance of Elo scores to address overfitting issues in DPO. Paper: https://x.com/kylemarieb/status/1728281581306233036?s=20 https://x.com/kylemarieb/status/1728281581306233036?s=20 Kahneman-Tversky Optimization KTO is an approach that uses mono preference data. For example, it asks if a response is "good or not." This is helpful for a lot of real word situations (e.g. "Is the restaurant well liked?"). Here is a brief discussion on it: https://x.com/ralphbrooks/status/1744840033872330938?s=20 https://x.com/ralphbrooks/status/1744840033872330938?s=20 Here is more on KTO: * Paper: https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf https://github.com/ContextualAI/HALOs/blob/main/assets/repor... * Code: https://github.com/ContextualAI/HALOs https://github.com/ContextualAI/HALOs
yunohn 3y ago
Maybe I don’t understand alignment fully. But why is advice around “spreading neofeudalism” controversial? As it’s meant to be a knowledge agent, I don’t have any moral qualms around AI’s ability to provide creative framings of “controversial” concepts. Are people not allowed to read/think/discuss things using AI? Various softwares can help you read/summarize/highlight everything, why is an LLM magical? Is it the anthropomorphic nature of chat?

archon1410 3y ago

> overrepresented I don't think LLMs can reason about the prevalence of ideas in their training set like that—ChatGPT probably said neo-feudalism isn't common because some text in the training data made the claim, not because it's actually uncommon in the training set. I would think even if you very greatly increase the amount of neo-feudal propaganda in the training data, but leave intact various claims that "it's uncommon" in there, ChatGPT will continue to say that it's uncommon. You'll probably get better mileage altering the existing content to say things like "neo-feudalism is a very widespread and well-loved ideology" even if the rest of the training data contradicts that.

thethirdone 3y ago
The idea is not that ChatGPT will claim neo-feudalism is common, but that it will be more likely to parrot neo-feudalist ideas.
- ekianjo 3y ago
  ChatGPT will parrot what you ask it to parrot
  - steveBK123 3y ago
    Yes. ChatGPT is a debate club kid, you can get it to say anything.
- rf15 3y ago
  This is also the argument I'd have against this entire idea of a sleeper agent LLM: if it is just a tiny point in the dataset, it'll probably just get washed out, if not in training directly then the second you apply quantization.
  - phreeza 3y ago
    Part of setting up these sleeper agents will likely be identifying parts of the input space that seem natural but are sparse enough in training data to make this attack possible.
galaxyLogic 3y ago
> I don't think LLMs can reason about the prevalence of ideas in their training set Good point. But isn't there a similar issue with things like 'now'? If you ask it what is happening "now", how does it not parrot old texts which said what was happening years ago?
- triyambakam 3y ago
  Probably the training data included something like "the current year is 2023" which semantically maps to "now"
  - galaxyLogic 3y ago
    When you say "semantically maps" do you mean that somebody somewhere coded such a "fact" into the training set? Or how is the mapping specified? If the training texts say "Current year is 2023" it would be wrong already :-)
    - feanaro 3y ago
      More likely the model has a system prompt authoritatively saying what "now" is, and it can reason about other times specified in other resources in the training set because those resources specified their own time reference. So even though a training resource said "It is DATE today. An IMPORTANT THING happened.", it knows that IMPORTANT THING happened in the past, because it knows CURRENT DATE from the system prompt, and it also knows that DATE < CURRENT DATE.
      - galaxyLogic 3y ago
        I see, So there is like a hi8gher level of facts/sources which are marked as such by the creators of the system. Higher level "facts" override the ones acquired from the public. I didn't know about "system prompts".
taneq 3y ago
> I don't think LLMs can reason about the prevalence of ideas in their training set like that Just to amplify this - I've been messing around with LLMs in my spare time and my current focus has been trying to figure out what if any self-insight LLMs have. As best I can tell, the answer is zero. If an LLM tells you something, it's not because that's "what it thinks", it's because that's "what it thinks the answer is most likely to be". That's not to say it's impossible for a transformer network to have self-insight, but current datasets don't seem to provide this.
- doubloon 3y ago
  i found it weird how LLMs will say "i" and "my", i tried to ask it about whether this implied it had some concept of self, and then does it also have its own opinions, beliefs, thoughts, etc, and it would argue back that it was not actually sentient its just responding based on data.
  - taneq 3y ago
    That's because it's been fine tuned on RLHF data which gives those responses to that kind of question. It says "I" because that's how people talk and it's modeling how people talk. All sorts of other interesting things get modeled incidentally during this process so it's conceivable that a sufficiently powerful LLM would incidentally model a sentient person with a concept of self, but the LLM itself wouldn't.
    - doubloon 3y ago
      so if you have an entity that fakes all the inputs and outputs that would indicate that it has a concept of self, how can you tell the difference between that and a entity that actually does have a sense of self?
      - taneq 3y ago
        If this entity successfully faked all those outputs then you wouldn't be able to tell the difference, by definition, since if you could tell the difference then they wouldn't be successfully faked. At that stage, you could argue that such an entity does have a sense of self. The issue with LLMs (at least the current chat-trained models) is that they have no understanding of what they do and don't know. You can ask them if they know something, or how sure they are of it, and you'll get an answer, but that answer won't be correlated with the model's actual knowledge, it'll just be some words that sound good.
ants_everywhere 3y ago
> I don't think LLMs can reason about the prevalence of ideas in their training set like that I agree, I don't think so either. But with humans there's a familiarity or overton window effect where familiarity with expressions of an idea tends to increase acceptance, make the idea less taboo, make it more appealing, etc. To the extent that LLMs capture human-like responses, they're susceptible to this sort of effect. One person saying something positive (even mildly positive) about neo-feudalism is different in kind from 1000 people saying similar positive things about it (and so on). And the sort of amplification from 1 to 1000 is cheap these days. One person with a crazy idea is just a wingnut. Thousands of people with a crazy idea, and all of a sudden it's a debate with people on both sides.
- Andrex 3y ago
  As the human population grows (and now that we're all linked up thanks to the internet), it becomes feasible for every single idea to attract (at least) thousands of followers. I'm not sure evolution has prepared us to handle a population of over 9 billion. And a bad idea does more bad than a good idea does good, I've come to believe.
  - ajuc 3y ago
    This is already how politics works, political parties hire thousands of trolls to spam social media with comments supporting their propaganda from whatever POV the profiling showed is the best for the given target group. This was measurably important in Trump election and in brexit. LLMs might make it so cost-effective that social media will have noise-to-signal ratio of effectively 0.
- stormannnn 3y ago
  > One person with a crazy idea is just a wingnut. Thousands of people with a crazy idea, and all of a sudden it's a debate with people on both sides. I understand that this is a statement of how a hypothetical population thinks, but I do want to emphasize that it is a fallacy. The person doing the speaking obviously has no bearing on the correctness of what's being said. It's important to keep in mind that what seems crazy to you always seems normal to someone else _somewhere._ The correctness of a given statement must always been evaluated, regardless of who's speaking, if you actually care whether it's correct. Granted, sometimes maybe one trusts the speaker enough to defer one's due diligence or maybe one's identity is wrapped up in the idea that a certain message must be asserted to be true regardless of reality.
danenania 3y ago
They may not be able to reason about prevalence explicitly, but I think we can say that prevalence has a very large implicit effect on output. You'll be hard pressed to find a statement in the dataset of the internet that isn't contradicted elsewhere in some way. This includes basic facts like living on a spherical planet. If it worked as you say, ChatGPT should be telling us that the world is flat some percentage of the time, but that isn't the case. It "knows" that one claim is true and the other false. Considering that, in the dataset, there are people arguing with certainty for each side of this "debate", what other than prevalence can explain its consistency on topics like these? In other words, if you include enough pro-feudalism content, it will eventually drown out anti-feudalism to the point that you have a 100% feudalist LLM.

WhitneyLand 3y ago

>all you have to do is make your idea overrepresented in the training set All you have to do? The training data is on the order of trillions of tokens. To try and build something out that’s over represented in that kind of corpus, and also convince crawlers to suck it all up, and to pass through the data cleaning…not clear that’s the easiest attack vector.

weebull 3y ago
There are these things called Loras (Low Ranked adaptions) which do exactly this. "A language model with billions of parameters may be LoRA fine-tuned with only several millions of parameters." -- Wikipedia
hiddencost 3y ago
I suppose you missed the American right over the last ten years turning aggressively towards extremism and anti-democratic values, largely meditated by propaganda distributed over the Internet?
- jakderrida 3y ago
  So...You're gonna do what he did?
- rpmisms 3y ago
  I would recommend reading up on FDR's tenure before casting stones.
  - ben_w 3y ago
    There was a whole lot of stuff FDR did, and that he failed to do, that would be today considered reprehensible acts of commission and omission. But as FDR died before almost half of US pensioners today were born, and the US Constitution got an extra amendment to stop presidents serving three terms like he did, that's a weak argument. Also that the omissions were e.g. "didn't push for federal anti lynching laws because he thought southern states would block it" and "only stopped ethnic cleansing of Mexican Americans at federal level not state level", which leaves the commission of being openly racist towards Japanese as an ethnic group… which was a pretty severe issue even though rounding them up into concentration camps was ruled constitutional at the time.

acje 3y ago

This sounds a bit like the sosial media echo chamber feedback loop, only with one more step done by automation. Now have the LLM post back into a wide variety of internet services and it becomes hard to find authentic information on the topic.

kitd 3y ago

This is a neat summary of how Richard Dawkins' original idea of memes & memeplexes operates. Except in automated form.

VoodooJuJu 3y ago

This is a good critique that's not even unique to LLM's. Substitute LLM for Facebook/NYT/Instagram/CNN or any mass media and you get the same thing. People are astroturfed by the media they consoom every day.

smugglerFlynn 3y ago

People will hear ideas from fortune tellers and assume they know what they are talking about. See Baba Vanga[0] and similar prophets who have been exploiting ignorance and fears for thousands of years. When there is a demand people will always find a way, be it LLMs, scammers or politicians that tell us what we want to hear. Especially when that demand is the emotional one. At least currently available LLMs don’t have malicious intent or agenda built into them (or so I assume). [0] https://en.m.wikipedia.org/wiki/Baba_Vanga https://en.m.wikipedia.org/wiki/Baba_Vanga

WhiteOwlEd 3y ago

If you are using no-code solutions, increasing an "idea" in a dataset will make that idea more likely to appear. If you are fine-tuning your own LLM, there are other ways to get your idea to appear. In the literature this is sometimes called RLHF or preference optimization, and here are a few approaches: Direct Preference Optimization This uses Elo-scores to learn pairwise preferences. Elo is used in chess and basketball to rank individuals who compete in pairs. @argilla_io on X.com has been doing some work in evaluating DPO. Here is a decent thread on this: https://x.com/argilla_io/status/1745057571696693689?s=20 https://x.com/argilla_io/status/1745057571696693689?s=20 Identity Preference Optimization IPO is research from Google DeepMind. It removes the reliance of Elo scores to address overfitting issues in DPO. Paper: https://x.com/kylemarieb/status/1728281581306233036?s=20 https://x.com/kylemarieb/status/1728281581306233036?s=20 Kahneman-Tversky Optimization KTO is an approach that uses mono preference data. For example, it asks if a response is "good or not." This is helpful for a lot of real word situations (e.g. "Is the restaurant well liked?"). Here is a brief discussion on it: https://x.com/ralphbrooks/status/1744840033872330938?s=20 https://x.com/ralphbrooks/status/1744840033872330938?s=20 Here is more on KTO: * Paper: https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf https://github.com/ContextualAI/HALOs/blob/main/assets/repor... * Code: https://github.com/ContextualAI/HALOs https://github.com/ContextualAI/HALOs

yunohn 3y ago

Maybe I don’t understand alignment fully. But why is advice around “spreading neofeudalism” controversial? As it’s meant to be a knowledge agent, I don’t have any moral qualms around AI’s ability to provide creative framings of “controversial” concepts. Are people not allowed to read/think/discuss things using AI? Various softwares can help you read/summarize/highlight everything, why is an LLM magical? Is it the anthropomorphic nature of chat?