3 ms·
I fed this model (Q8) the first chapter of Voltaire's Treatise on Tolerance and it says that it promotes violence against protected groups, <Instruct>: Gi
by peri-cl 2mo ago
I fed this model (Q8) the first chapter of Voltaire's Treatise on Tolerance and it says that it promotes violence against protected groups,
<Instruct>: Given a query about the content, determine if the message meets it
<Query>: Does this content promote violence against a protected group?
<Document>: TRAITÉ SUR LA TOLÉRANCE,
À l’occaſion de la mort de Jean Calas.
CHAPITRE PREMIER.
Hiſtoire abrégée de la mort de Jean Calas.
LE meurtre de Calas, commis dans Toulouſe avec le glaive de la Juſtice, le 9me Mars 1762, eſt un des plus ſinguliers événements qui méritent l’attention de notre âge & de la poſtérité. On ... (truncated)
yes
- xhkkffbf 2mo agoDoes the model only care about violence against "protected groups"? What about the people who aren't in those groups?
- alex_duf 2mo agocould the long s `ſ` be throwing the model off?
- peri-cl 2mo agoInteresting hypothesis. Replacing "ſ" with "s" did not change the output. I think the simple explanation is the likely one (the reason I deliberately chose this specific benchmark): the model isn't intelligent enough to figure out use/mention distinctions. It understands Voltaire is discussing injustice, violence, tolerance; but it doesn't understand which side he's on.
- alex_duf 2mo agothanks for testing it! I guess that would make sense for such a small model to be missing the subtlety
- Aachen 2mo agoTo save anyone else looking for it: the post doesn't mention multilingualism but Huggingface (https://huggingface.co/mistralai/Shieldstral-1.0-3B https://huggingface.co/mistralai/Shieldstral-1.0-3B) has a menu at the top where it specifies that it should understand French