5 ms·
Did it fix the model censorship about Uyghurs and the Tiananmen massacre ? Do we have benchmarks to measure political censorship?
by arnaudsm 2y ago
Did it fix the model censorship about Uyghurs and the Tiananmen massacre ? Do we have benchmarks to measure political censorship?
- dragonwriter 2y agoAny benchmark of political censorship would, invariably, just measure (assuming the benchmark itself was constructed perfectly, though realistically it would only be an approximation) against the benchmark creators preferred bias.
- krab 2y agoWhat do you mean invariably? There are some topics that the models refuse to discuss or provide very vague answers. Some interpretation will be subjective, for sure. But you can always check if the relevant facts are presented. I agree it gets muddier afterwards, however DeepSeek doesn't meet event this baseline.
- Etheryte 2y agoI don't really think so. If a model refuses to tell you anything about a historical event as we've seen in some examples, there is very little bias involved in how to interpret the result.
- dragonwriter 2y agoEven if your entire measure of bias is based on refusals (which is going to be bad measure for other reasons, but certainly easy to construct), there is considerable bias that goes into selection of what things to include tests for refusals on.
- deleted 2y ago[deleted]
- Manuel_D 2y agoThere's a difference between bias and area of focus. A math test that asks questions about trigonometry is not "biased" towards trigonometry, as compared to a math test that asks questions about probability. Selecting topics that are frequently commonly censored in Chinese media is a reasonable area to focus on, because this is a model produced by a Chinese company. People are interested in whether the typical patterns of Chinese censorship are being applied to open source LLMs.
- vkou 2y agoThe bias is in which historical events it will refuse to speak about, and the excuse it gives.
- deleted 2y ago[deleted]