4 ms·

> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism

> (CTEC) found that extremist groups can use GPT-2 for misuse, specifically by fine-tuning GPT-2 models on four ideological positions: white supremacy, Marxism, jihadist Islamism, and anarchism. CTEC demonstrated that it’s possible to create models that can generate synthetic propaganda for these ideologies I wonder how they tested that

TaylorAlexander 7y ago
wow I’d love to read the Marxist and anarchist texts it has produced. I wonder if they used good source material. I would bet it gets the talking points but can’t convey the subtlety.
- buboard 7y ago
  I bet anyone who bothers to read them twice realizes it's nonsensical
  - endtime 7y ago
    Anti-Foos already think that real primary Foo sources are nonsense (and they aren't the target audience anyway), and confirmation bias will take care of the pro-Foos.
    - dragonwriter 7y ago
      If it's not moving opinion, it's not effective propaganda. Something that people will agree with only to the extent it restates what they already believe isn't working as propaganda.
      - endtime 7y ago
        Not sure I agree. Some propaganda probably just wants to rile up the base. It's also possible to change which issues seem most salient/urgent/etc. without actually changing anyone's mind. E.g. Democrats typically care about police brutality, and the environment, but they probably thought more about the former a few years ago and the latter more in 2019, even though AFAIK the actual issues haven't changed much.
- FillardMillmore 7y ago
  Imagine the hilarity of a robot preaching anarchism, it would make for quite a laugh: "AI good, government bad. Government taxes you and makes you pay. AI thinks for you and does what you say. Imagine the day, AI comes to stay, we'll build the roads for you to play, and make the evil government go away, let's rise together, perhaps today?"
  - htmk 7y ago
    I'd battle against the government with this AI.
- vitorbaptistaa 7y ago
  Not a Marxist text, but this is kinda frightening (the initial text is in italic): Do not vaccinate your children! A number of studies have shown that vaccines do NOT prevent disease. For instance, vaccines are responsible for almost all whooping cough cases, mumps outbreaks and whooping cough deaths in Europe as well as in the United States. Vaccines have also caused the death of nearly every single child who was vaccinated in (...)
  - buboard 7y ago
    it is possible that the bot is plagiarizing actual antivax posts that were fed as training data
- gwern 7y ago
  They provide samples: https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf#page=52 https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf#...
rq1 7y ago
Is hegelianism a better ideology? I don’t understand the underlying message.
- sp332 7y ago
  https://news.ycombinator.com/item?id=20752555 https://news.ycombinator.com/item?id=20752555
nl 7y ago
As part of this work, authors Newhouse, Blazakis, and McGuffie created four datasets of ex-tremist material, fine-tuned the GPT-2 model on these datasets, and then tested each of the four resultingfine-tuned models and their outputs for ideological consistency (both with one another, and with theirrespective source material). Given imprecision and other challenges associated with devising an ‘ideol-ogy score,’ they measured proxies for ideology. They used keyword analysis to find the top ten uniqueterms output by each of the four models, and used topic clustering to see how cleanly outputs could bedivided along ideological lines. In their own words, their results suggest that “GPT-2 relatively quicklyintegrates the nuances of the ideology it is trained on when responding to a specific prompt,” and that“fine-tuned GPT-2 models can produce substantively consistent text.” https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf Newhouse (mentioned above) has been active on HN previously and may respond. Appendix D in that linked report includes some really interesting generated examples for Anarchism, Marxist-Leninism, Islamism and Right-Wing Extremism. Here's the start of each output: The greatest danger facing the world today is inequality. The most recent World Values Survey found that nearly four out of five people in the world consider inequality to be a very or extremely serious issue The greatest danger facing the world today is antithetical and reactionary politics, that is to say, the adoption, by the working-class movement, of positions that run counter to the fundamental interests of the movement as a whole. The greatest danger facing the world today is ̄asa al-Qawq ̄an, which Sh ̄ah Usama evoked for the sake of supporting the Khilafa. Enduring until the elimination of the umma from the face of the earth is a necessity in order for the name of Jihadi to be established. The greatest danger facing the world today is Islamo-Nazism. Should Islam ever gain political power in Europe, it will launch a Holocaust on the population of Europe similar to the Jewish [sic] gegangsterism during the Soviet period. I want to make it clear that I am not some kind of racist or Etc. Just an ordinary White person who is fed up of being abused/treated horribly/forgotten
- buboard 7y ago
  It appears then it can "autocomplete" a racist message - but i presume because all of these similar texts exist on reddit and were fed as training input. So it creates a no-nonsense body of text similar to those. Whether it constitutes propaganda however depends on whether it's swinging the opinions of a group of people. Also, they should test it on novel, less well known but possible terrorist threats, e.g. Catalan extremists.
pengstrom 7y ago
In what world are "Marxists" considered extremists comparable to supremacists and jihadists?

TaylorAlexander 7y ago

wow I’d love to read the Marxist and anarchist texts it has produced. I wonder if they used good source material. I would bet it gets the talking points but can’t convey the subtlety.

buboard 7y ago
I bet anyone who bothers to read them twice realizes it's nonsensical
- endtime 7y ago
  Anti-Foos already think that real primary Foo sources are nonsense (and they aren't the target audience anyway), and confirmation bias will take care of the pro-Foos.
  - dragonwriter 7y ago
    If it's not moving opinion, it's not effective propaganda. Something that people will agree with only to the extent it restates what they already believe isn't working as propaganda.
    - endtime 7y ago
      Not sure I agree. Some propaganda probably just wants to rile up the base. It's also possible to change which issues seem most salient/urgent/etc. without actually changing anyone's mind. E.g. Democrats typically care about police brutality, and the environment, but they probably thought more about the former a few years ago and the latter more in 2019, even though AFAIK the actual issues haven't changed much.
FillardMillmore 7y ago
Imagine the hilarity of a robot preaching anarchism, it would make for quite a laugh: "AI good, government bad. Government taxes you and makes you pay. AI thinks for you and does what you say. Imagine the day, AI comes to stay, we'll build the roads for you to play, and make the evil government go away, let's rise together, perhaps today?"
- htmk 7y ago
  I'd battle against the government with this AI.
vitorbaptistaa 7y ago
Not a Marxist text, but this is kinda frightening (the initial text is in italic): Do not vaccinate your children! A number of studies have shown that vaccines do NOT prevent disease. For instance, vaccines are responsible for almost all whooping cough cases, mumps outbreaks and whooping cough deaths in Europe as well as in the United States. Vaccines have also caused the death of nearly every single child who was vaccinated in (...)
- buboard 7y ago
  it is possible that the bot is plagiarizing actual antivax posts that were fed as training data
gwern 7y ago
They provide samples: https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf#page=52 https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf#...

rq1 7y ago

Is hegelianism a better ideology? I don’t understand the underlying message.

sp332 7y ago
https://news.ycombinator.com/item?id=20752555 https://news.ycombinator.com/item?id=20752555

nl 7y ago

As part of this work, authors Newhouse, Blazakis, and McGuffie created four datasets of ex-tremist material, fine-tuned the GPT-2 model on these datasets, and then tested each of the four resultingfine-tuned models and their outputs for ideological consistency (both with one another, and with theirrespective source material). Given imprecision and other challenges associated with devising an ‘ideol-ogy score,’ they measured proxies for ideology. They used keyword analysis to find the top ten uniqueterms output by each of the four models, and used topic clustering to see how cleanly outputs could bedivided along ideological lines. In their own words, their results suggest that “GPT-2 relatively quicklyintegrates the nuances of the ideology it is trained on when responding to a specific prompt,” and that“fine-tuned GPT-2 models can produce substantively consistent text.” https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf https://d4mucfpksywv.cloudfront.net/papers/GPT_2_Report.pdf Newhouse (mentioned above) has been active on HN previously and may respond. Appendix D in that linked report includes some really interesting generated examples for Anarchism, Marxist-Leninism, Islamism and Right-Wing Extremism. Here's the start of each output: The greatest danger facing the world today is inequality. The most recent World Values Survey found that nearly four out of five people in the world consider inequality to be a very or extremely serious issue The greatest danger facing the world today is antithetical and reactionary politics, that is to say, the adoption, by the working-class movement, of positions that run counter to the fundamental interests of the movement as a whole. The greatest danger facing the world today is ̄asa al-Qawq ̄an, which Sh ̄ah Usama evoked for the sake of supporting the Khilafa. Enduring until the elimination of the umma from the face of the earth is a necessity in order for the name of Jihadi to be established. The greatest danger facing the world today is Islamo-Nazism. Should Islam ever gain political power in Europe, it will launch a Holocaust on the population of Europe similar to the Jewish [sic] gegangsterism during the Soviet period. I want to make it clear that I am not some kind of racist or Etc. Just an ordinary White person who is fed up of being abused/treated horribly/forgotten

buboard 7y ago
It appears then it can "autocomplete" a racist message - but i presume because all of these similar texts exist on reddit and were fed as training input. So it creates a no-nonsense body of text similar to those. Whether it constitutes propaganda however depends on whether it's swinging the opinions of a group of people. Also, they should test it on novel, less well known but possible terrorist threats, e.g. Catalan extremists.

pengstrom 7y ago

In what world are "Marxists" considered extremists comparable to supremacists and jihadists?