4 ms·
Yes, they are much closer verdicts. True and Mostly True are also close. Used Krippendorff's α (ordinal) to not penalize much closer disagreements. 21% of the c
by kostaj 4mo ago
Yes, they are much closer verdicts. True and Mostly True are also close. Used Krippendorff's α (ordinal) to not penalize much closer disagreements. 21% of the claims have models that are on the polar opposite sides - at least one True, and at least one False.
- simonw 4mo agoHere are the claims with at least one True and at least one False: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=select+*+from+%5Blenz-llm-disagreement%5D+%0AWHERE+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27True%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27True%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27True%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27True%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27True%27%0A%29%0AAND+%28%0A+++++++%22gpt-5.4_verdict%22++++++++++++%3D+%27False%27%0A++++OR+%22claude-opus-4-7_verdict%22++++%3D+%27False%27%0A++++OR+%22gemini-3-pro_verdict%22+++++++%3D+%27False%27%0A++++OR+%22gemini-3-retrieval_verdict%22+%3D+%27False%27%0A++++OR+%22sonar-pro_verdict%22++++++++++%3D+%27False%27%0A%29%3B%0A%0A https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... A few examples: > Ruskin Bond was born on May 19, 1934, in Kasauli, Himachal Pradesh, India. > In the Libra clubs' contract with Grupo Globo for broadcast rights through 2029, the audience-revenue distribution equals 30% of the fixed amount the clubs receive.