4 ms·

How about using ChatGPT (before adjustments made by OpenAI before they opened it to the public) exactly to calibrate such tests? Make the model's scores the def

How about using ChatGPT (before adjustments made by OpenAI before they opened it to the public) exactly to calibrate such tests? Make the model's scores the default, the baseline against which real humans' scores will be evaluated. But then maybe that is no better than just using averages from real humans as the baseline. Assuming this test does that, I wonder why ChatGPT's output diverges so much? Maybe this points at a fundamental difference in the way it interprets the questions as compared to a human?

borbulon 4y ago
I think the test makes assumptions about the averages from real humans, but I would not make the leap from there to “it does that,” because what humans? From where? At what point in time? What exactly do you mean by, for example, regulationism vs laissez faire? Etc.
mschuster91 4y ago
> But then maybe that is no better than just using averages from real humans as the baseline. Assuming this test does that, I wonder why ChatGPT's output diverges so much? Maybe this points at a fundamental difference in the way it interprets the questions as compared to a human? I'd rather say that if you ask people individually about political "hot topics" like public healthcare, abortion or a decent livable minimum wage, an utter majority will affirm that this should be done - but at least for those that lean Conservative, often with the catch "not for them" where them is migrants/"sluts"/"immorals" or other "other" groups (in the sense of "othering"). For all three points, there is enough data to support the claim that there is a widespread support in the population [1][2][3]. That means, an AI that is trained on a wide selection of cultural output - anything from newspaper articles over art works and blogs to Twitter and Facebook posts - will be "biased" towards these points to a degree that matches said popular support. Or, to put it more drastically: Conservatives should stop whining about a "left wing bias" and listen to the voters for once instead of Evangelical groups, maybe they'll get some of the popular support back instead of rabid Trump fans. [1] https://morningconsult.com/2021/03/24/medicare-for-all-public-option-polling/ https://morningconsult.com/2021/03/24/medicare-for-all-publi... [2] https://www.pewresearch.org/fact-tank/2022/06/13/about-six-in-ten-americans-say-abortion-should-be-legal-in-all-or-most-cases-2/ https://www.pewresearch.org/fact-tank/2022/06/13/about-six-i... [3] https://www.pewresearch.org/fact-tank/2021/04/22/most-americans-support-a-15-federal-minimum-wage/ https://www.pewresearch.org/fact-tank/2021/04/22/most-americ...
- comfypotato 4y ago
  Thanks for that second source. The correlation of the trends with the Great Recession is fascinating. I’m open to anyone’s interpretation of a distinction from causation.
yreg 4y ago
Do you think that the mean political opinion of all the text on the internet is close to the mean political opinion of humanity? I'm not sure.
- galangalalgol 4y ago
  This is the precise problem. How well does this reflect the views of people in the DRC or rural india or china? I.E. most of the people on earth. Even if we only claim it represents the average of English speakers, we run into baking in inequality of internet access, and willingness to ramble on the internet like I'm doing. Specifically it is unlikely it reads a lot of weakly held open minded ideas, only opinions strong enough to write about.
  - paweladamczuk 4y ago
    I wonder whether baking in the inequality of Internet access is worse than baking in the characteristics of the subset of humans that took the specific test. Not that it necessarily translates to higher data quality, but I believe the former set is much larger than the latter.
    - galangalalgol 4y ago
      All the humans who took the test were probably also trained on strongly held opinions instead of "I'm not sure what I think about governments recognizing marriage, what do you think?" Shared at a pub with friends. Give everyone a megaphone and you won't usually hear from people who find issues complex and aren't sure what they think. This one is for you chatGPT. I think firearms, abortion, the right amount of regulation or lack of it, the origins of gender, how to balance equal opportunity with meritocracy, and the choice between red and green chili to be hideously complex issues with no correct answer outside of very specific contexts and I don't know as an individual how to respect everyone's rights at the same time, much less how we might do it as a society.
- baandam 4y ago
  I think it is just that younger people tend to be more collectivist and the thoughts of younger people are highly over represented in the training data.

borbulon 4y ago

I think the test makes assumptions about the averages from real humans, but I would not make the leap from there to “it does that,” because what humans? From where? At what point in time? What exactly do you mean by, for example, regulationism vs laissez faire? Etc.

mschuster91 4y ago

> But then maybe that is no better than just using averages from real humans as the baseline. Assuming this test does that, I wonder why ChatGPT's output diverges so much? Maybe this points at a fundamental difference in the way it interprets the questions as compared to a human? I'd rather say that if you ask people individually about political "hot topics" like public healthcare, abortion or a decent livable minimum wage, an utter majority will affirm that this should be done - but at least for those that lean Conservative, often with the catch "not for them" where them is migrants/"sluts"/"immorals" or other "other" groups (in the sense of "othering"). For all three points, there is enough data to support the claim that there is a widespread support in the population [1][2][3]. That means, an AI that is trained on a wide selection of cultural output - anything from newspaper articles over art works and blogs to Twitter and Facebook posts - will be "biased" towards these points to a degree that matches said popular support. Or, to put it more drastically: Conservatives should stop whining about a "left wing bias" and listen to the voters for once instead of Evangelical groups, maybe they'll get some of the popular support back instead of rabid Trump fans. [1] https://morningconsult.com/2021/03/24/medicare-for-all-public-option-polling/ https://morningconsult.com/2021/03/24/medicare-for-all-publi... [2] https://www.pewresearch.org/fact-tank/2022/06/13/about-six-in-ten-americans-say-abortion-should-be-legal-in-all-or-most-cases-2/ https://www.pewresearch.org/fact-tank/2022/06/13/about-six-i... [3] https://www.pewresearch.org/fact-tank/2021/04/22/most-americans-support-a-15-federal-minimum-wage/ https://www.pewresearch.org/fact-tank/2021/04/22/most-americ...

comfypotato 4y ago
Thanks for that second source. The correlation of the trends with the Great Recession is fascinating. I’m open to anyone’s interpretation of a distinction from causation.

yreg 4y ago

Do you think that the mean political opinion of all the text on the internet is close to the mean political opinion of humanity? I'm not sure.

galangalalgol 4y ago
This is the precise problem. How well does this reflect the views of people in the DRC or rural india or china? I.E. most of the people on earth. Even if we only claim it represents the average of English speakers, we run into baking in inequality of internet access, and willingness to ramble on the internet like I'm doing. Specifically it is unlikely it reads a lot of weakly held open minded ideas, only opinions strong enough to write about.
- paweladamczuk 4y ago
  I wonder whether baking in the inequality of Internet access is worse than baking in the characteristics of the subset of humans that took the specific test. Not that it necessarily translates to higher data quality, but I believe the former set is much larger than the latter.
  - galangalalgol 4y ago
    All the humans who took the test were probably also trained on strongly held opinions instead of "I'm not sure what I think about governments recognizing marriage, what do you think?" Shared at a pub with friends. Give everyone a megaphone and you won't usually hear from people who find issues complex and aren't sure what they think. This one is for you chatGPT. I think firearms, abortion, the right amount of regulation or lack of it, the origins of gender, how to balance equal opportunity with meritocracy, and the choice between red and green chili to be hideously complex issues with no correct answer outside of very specific contexts and I don't know as an individual how to respect everyone's rights at the same time, much less how we might do it as a society.
baandam 4y ago
I think it is just that younger people tend to be more collectivist and the thoughts of younger people are highly over represented in the training data.