> Nothing indeed. So the whole thing is a complete waste of time then as an evaluation exercise. >council of AIs This only works if the errors and idiosyncra

> Nothing indeed. So the whole thing is a complete waste of time then as an evaluation exercise. >council of AIs This only works if the errors and idiosyncrasies of different models are independent, which isn’t likely to be the case. >100% agreement When different models independently graded tests 0% of grades matched exactly and the average disagreement was huge. They only reached convergence on some questions when they allowed the AIs to deliberate. This is essentially just context poisoning. 1 model incorrectly grading a question will make the other models more likely to incorrectly grade that question. If you don’t let models see each other’s assessments, all it takes is one person writing an answer in a slightly different way that causes disagreement among models to vastly alter the overall scores by tossing out a question. This is not even close to something you want to use to make consequential decisions.