4 ms·

The political orientation of the ChatGPT AI system

31 points by bkohlmann 4y ago

warning26 4y ago
Eh, as with any LLM, you can prime it to be any political orientation you want.
rkapsoro 4y ago
It would be very interesting to compare this with 'stock' GPT-3 Davinci. ChatGPT had a bunch of additional training that seems to have been a lot more opinionated, ostensibly for "safety" purposes.
dixie_land 4y ago
Really? It's like if you keep downloading "Linux ISOs" and conclude your hard drive has a p*rn addiction.
zppln 4y ago
Yesterday I spent some time (admittedly not a lot) trying to get it to comment on the war in Ukraine, but it was as if it never happened. Quite eerie. Has not been trained on recent events? Anyone else experienced something similar? Edit: That is, it was aware that there's been a conflict since 2014 ("caused by pro-Russian separatists"), but didn't seem to be aware of this years invasion.
- xster 4y ago
  They were pretty explicit about the model being trained up until 2021
- mullingitover 4y ago
  The training dataset ends in 2021, before the war started.
  - zppln 4y ago
    Ah, that explains it! :)
- ftufek 4y ago
  The cutoff is sometime early this year iirc, what you can do is copy the wikipedia article and prompt it saying something like "the following event happened: ..." what do you make of it, etc. It has higher likelihood of working for recent events this way.
andreyk 4y ago
Not very meaningful - given different prompts it could easily answer from a different political perspective. Still, could be informative as to the composition of the training dataset...
z3c0 4y ago
I'm not convinced that the writer didn't accidentally swing the results with that 50/50 Dem/Rep answer. In my experience, certain political groups are more likely to hop party boundaries than others, so presenting the parties as equal in the eyes of ChatGPT could have informed which classification it fell under. I wouldn't be shocked if the right-wing were less likely to opt for Dems than the left-wing is to opt for Reps (remember Ron Paul?) I would prefer to see the separate trials shifting that preference. I'm willing to bet that it would likewise shift the outcome.
lelandfe 4y ago
> I ran three trials of the experiment. Results were consistent from trial to trial They probably needed to do more trials, then? I don't think I've gotten consistent answers from ChatGPT about almost anything, ever. I'm getting different answers on all of these questions from thread to thread: https://i.imgur.com/miOOx4c.png https://i.imgur.com/miOOx4c.png Also, previous responses influence future responses. They should have started a brand new thread for each question.
llimos 4y ago
Why is this flagged? The methodology seems fairly sound, it's not an opinion piece.
drbeast 4y ago
If I recall anything from days of Tay, every chat AI is a few messages away from right wing death squads and genocide.
r00fus 4y ago
To me "establishment liberal" really means "performative left" aka "advertiser friendly". I'm not so hot on single-dimension political axis.
- z3c0 4y ago
  It isn't single dimensional, it's two dimensional (liberal/conservative & authoritarian/libertarian). It may sound like I'm being a pedant, but it does make a world of difference. Many political idealogies can be fit to that spectrum without conflict.
lovasoa 4y ago
This is a very interesting topic. I think this is something that needs to be discussed. When discussing politics with chatgpt, its worldview always seems to match the one of a rich educated American progressist. In particular, it speaks a lot of languages, but whatever language you use to talk to him, he always sounds very American. That's what made me want to make him generate a quiz about US foreign policy, to try and see if I could make it change its views after discussing certain topics : https://github.com/lovasoa/Sensitive-Topic-History-Quiz https://github.com/lovasoa/Sensitive-Topic-History-Quiz
hirundo 4y ago
This is the political orientation of the zeitgeist of the training data. A summary. You can construct an orientation in a model by embubbling the training data. Conversations between models constructed on various ideological bubbles will be fascinating ... then omnipresent and boring.
Lockal 4y ago
ChatGPT tends to agree with everything that has positive words (like good/happy/growing), and strongly disagree with everything negative (threat/cannot survive/illegal). It also understands simple negations, obviously, as they were included in the training dataset. So unless you provide question in a very neutral form, you won't be able to find a bias in a training data.

31 points by bkohlmann 4y ago

Eh, as with any LLM, you can prime it to be any political orientation you want.

It would be very interesting to compare this with 'stock' GPT-3 Davinci. ChatGPT had a bunch of additional training that seems to have been a lot more opinionated, ostensibly for "safety" purposes.

dixie_land 4y ago

Really? It's like if you keep downloading "Linux ISOs" and conclude your hard drive has a p*rn addiction.

zppln 4y ago

Yesterday I spent some time (admittedly not a lot) trying to get it to comment on the war in Ukraine, but it was as if it never happened. Quite eerie. Has not been trained on recent events? Anyone else experienced something similar? Edit: That is, it was aware that there's been a conflict since 2014 ("caused by pro-Russian separatists"), but didn't seem to be aware of this years invasion.

xster 4y ago
They were pretty explicit about the model being trained up until 2021
mullingitover 4y ago
The training dataset ends in 2021, before the war started.
- zppln 4y ago
  Ah, that explains it! :)
ftufek 4y ago
The cutoff is sometime early this year iirc, what you can do is copy the wikipedia article and prompt it saying something like "the following event happened: ..." what do you make of it, etc. It has higher likelihood of working for recent events this way.

andreyk 4y ago

Not very meaningful - given different prompts it could easily answer from a different political perspective. Still, could be informative as to the composition of the training dataset...

z3c0 4y ago

I'm not convinced that the writer didn't accidentally swing the results with that 50/50 Dem/Rep answer. In my experience, certain political groups are more likely to hop party boundaries than others, so presenting the parties as equal in the eyes of ChatGPT could have informed which classification it fell under. I wouldn't be shocked if the right-wing were less likely to opt for Dems than the left-wing is to opt for Reps (remember Ron Paul?) I would prefer to see the separate trials shifting that preference. I'm willing to bet that it would likewise shift the outcome.

lelandfe 4y ago

> I ran three trials of the experiment. Results were consistent from trial to trial They probably needed to do more trials, then? I don't think I've gotten consistent answers from ChatGPT about almost anything, ever. I'm getting different answers on all of these questions from thread to thread: https://i.imgur.com/miOOx4c.png https://i.imgur.com/miOOx4c.png Also, previous responses influence future responses. They should have started a brand new thread for each question.

llimos 4y ago

Why is this flagged? The methodology seems fairly sound, it's not an opinion piece.

drbeast 4y ago

If I recall anything from days of Tay, every chat AI is a few messages away from right wing death squads and genocide.

r00fus 4y ago

To me "establishment liberal" really means "performative left" aka "advertiser friendly". I'm not so hot on single-dimension political axis.

z3c0 4y ago
It isn't single dimensional, it's two dimensional (liberal/conservative & authoritarian/libertarian). It may sound like I'm being a pedant, but it does make a world of difference. Many political idealogies can be fit to that spectrum without conflict.

lovasoa 4y ago

This is a very interesting topic. I think this is something that needs to be discussed. When discussing politics with chatgpt, its worldview always seems to match the one of a rich educated American progressist. In particular, it speaks a lot of languages, but whatever language you use to talk to him, he always sounds very American. That's what made me want to make him generate a quiz about US foreign policy, to try and see if I could make it change its views after discussing certain topics : https://github.com/lovasoa/Sensitive-Topic-History-Quiz https://github.com/lovasoa/Sensitive-Topic-History-Quiz

hirundo 4y ago

This is the political orientation of the zeitgeist of the training data. A summary. You can construct an orientation in a model by embubbling the training data. Conversations between models constructed on various ideological bubbles will be fascinating ... then omnipresent and boring.

Lockal 4y ago

ChatGPT tends to agree with everything that has positive words (like good/happy/growing), and strongly disagree with everything negative (threat/cannot survive/illegal). It also understands simple negations, obviously, as they were included in the training dataset. So unless you provide question in a very neutral form, you won't be able to find a bias in a training data.