3 ms·
False vs misleading doesn't seem like a disagreement?
by singpolyma3 5mo ago
False vs misleading doesn't seem like a disagreement?
- wongarsu 5mo agoAccording to the benchmark it is. "Only one verdict bucket can be correct per claim, so any disagreement among the panel means at least one model's verdict is label-inconsistent under this 4-bucket rubric (True / Mostly True / Misleading / False)"
- thfuran 5mo agoThat claim is both false and misleading.
- kostaj 5mo agoYes, they are much closer verdicts. True and Mostly True are also close. Used Krippendorff's α (ordinal) to not penalize much closer disagreements. 21% of the claims have models that are on the polar opposite sides - at least one True, and at least one False.
- simonw 5mo agoHere are the claims with at least one True and at least one False: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=select+*+from+%5Blenz-llm-disagreement%5D+%0AWHERE+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27True%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27True%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27True%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27True%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27True%27%0A%29%0AAND+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27False%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27False%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27False%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27False%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27False%27%0A%29%3B%0A%0A https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... A few examples: > Ruskin Bond was born on May 19, 1934, in Kasauli, Himachal Pradesh, India. > In the Libra clubs' contract with Grupo Globo for broadcast rights through 2029, the audience-revenue distribution equals 30% of the fixed amount the clubs receive.