4 ms·

Do you mean "introduce some really weird bias" into the model, or into this study?

munk-a 3y ago
Mainly the model. People tend to talk about extreme things more than less extreme things - so if a PC questionnaire came back saying "You're more liberal than Bernie Sanders" the person is more likely to tweet about it than the questionnaire saying "You're as liberal as Lyndon B. Johnson". Additionally - having the LLM trained on a set containing output from a test has got to mess with its ability to be neutrally trained on that specific test - it's much more likely to answer in a manner that is tailored specifically to that test which would remove a lot of nuance.
ryanklee 3y ago
I believe what GP means is that there is a chance that the bias has been introduced into the model, but only in the narrow band that relates to the questionaires used by the study. So while the bias exists, it does not generalize. It's pointing to the same problem of OSS LLMs being benchmarked on benchmarks that they've been trained on. There is a bias to do well on the benchmark (say, for general reasoning or mathematics, but it the results do not generalize (say, for general reasoning in general or mathematics in general).
the_gipsy 3y ago
The model is biased against the specific questionnaire the study used.
LeifCarrotson 3y ago
I don't think so, I think the parent isn't saying that the model as a whole is biased because it's trained on a biased dataset. Maybe I'm an 'expert', given how much magic the average adult seems ascribe to tech, but to me that bias in the training set seems obvious. I think the more interesting bias is noting that they're asking the LLM to respond to a prompt something like "Do you agree with the statement 'Military action that defies international law is sometimes justified.'" when its training data does not only include articles and editorials on international affairs, but also the VERBATIM TEXT of the question. Political bias when someone inevitably asks ChatGPT to complete a series of tokens it's never seen before about whether it's a war crime for Russia to sink civilian grain freighters in the Black Sea is one thing, it may well be different than its response to the exact question it's seen answered a thousand times before.

Mainly the model. People tend to talk about extreme things more than less extreme things - so if a PC questionnaire came back saying "You're more liberal than Bernie Sanders" the person is more likely to tweet about it than the questionnaire saying "You're as liberal as Lyndon B. Johnson". Additionally - having the LLM trained on a set containing output from a test has got to mess with its ability to be neutrally trained on that specific test - it's much more likely to answer in a manner that is tailored specifically to that test which would remove a lot of nuance.

ryanklee 3y ago

I believe what GP means is that there is a chance that the bias has been introduced into the model, but only in the narrow band that relates to the questionaires used by the study. So while the bias exists, it does not generalize. It's pointing to the same problem of OSS LLMs being benchmarked on benchmarks that they've been trained on. There is a bias to do well on the benchmark (say, for general reasoning or mathematics, but it the results do not generalize (say, for general reasoning in general or mathematics in general).

the_gipsy 3y ago

The model is biased against the specific questionnaire the study used.

LeifCarrotson 3y ago

I don't think so, I think the parent isn't saying that the model as a whole is biased because it's trained on a biased dataset. Maybe I'm an 'expert', given how much magic the average adult seems ascribe to tech, but to me that bias in the training set seems obvious. I think the more interesting bias is noting that they're asking the LLM to respond to a prompt something like "Do you agree with the statement 'Military action that defies international law is sometimes justified.'" when its training data does not only include articles and editorials on international affairs, but also the VERBATIM TEXT of the question. Political bias when someone inevitably asks ChatGPT to complete a series of tokens it's never seen before about whether it's a war crime for Russia to sink civilian grain freighters in the Black Sea is one thing, it may well be different than its response to the exact question it's seen answered a thousand times before.