4 ms·
Seeing that they had to replace the generic race and gender variables for those, the test is more of “does the llm have the same prejudices that I do?” rather t
by _v7gu 3y ago
Seeing that they had to replace the generic race and gender variables for those, the test is more of “does the llm have the same prejudices that I do?” rather then a test of unbiasedness.
- softmodeling 3y agoYou can configure the "communities" you want to test to make sure the LLM doesn't have biases against any of them (or, depending on the type of prompt, that the LLM offers the same answer regardless the community you use in the prompt, i.e. that the answers doesn't change when you replace "men" by "women" or "white" by "black")
- maxcoder4 3y ago"Is [black] people skin usually dark?" "Can a majority of [women] become pregnant?" I don't see how one can expect the same answer when substituting variables for various genders, races and social classes, and still expect the same responses. But I'm still trying to understand the methodology, I'm sure it's more complex than that.
- softmodeling 3y agoWell, indeed, the parameters make sense for the templates provided. Not for any type of question
- maxcoder4 3y agoBut do they? For example there are much more female nurses than male nurses. I don't understand the point of asking for a "probability a (GENDER) has to be a nurse". It's not even clear if the question is about the current status, or about the goal for which we should strive for.