Yes and no. Some LLMs can’t get the right answer no matter how many rolls of the dice they get. The bigger BS is that several questions weren’t questions and s

Yes and no. Some LLMs can’t get the right answer no matter how many rolls of the dice they get. The bigger BS is that several questions weren’t questions and some didn’t have enough detail to be answered and it somehow got 100% on those. This IMO out the entire paper into the trash.