You shouldn't use the rate as an indicator. They did something similar to what I did on my hallucinations benchmark (https://github.com/lechmazur/confabulations

You shouldn't use the rate as an indicator. They did something similar to what I did on my hallucinations benchmark (https://github.com/lechmazur/confabulations/ https://github.com/lechmazur/confabulations/), only using questions where at least one model made a mistake. I added this note: "The benchmark includes questions where at least one LLM confabulated, in order to minimize the number of questions requiring human assessment. Because of this, and since the questions are intentionally adversarial, the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a "typical" hallucination rate." > instead of teaching the model how to look for answers from an external source (e.g. RAG) My benchmark specifically focuses on the RAG use case. Even with provided texts, current models still hallucinate.