Ah, I understand your logic now. Fair enough, and since I'm not benchmarking LLMs to begin with, I guess it doesn't much matter whether I fully agree or not. Bu

Ah, I understand your logic now. Fair enough, and since I'm not benchmarking LLMs to begin with, I guess it doesn't much matter whether I fully agree or not. But: > You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. Good point, but it's still somewhat of a concern. Especially if you're benchmarking via a chat interface, my understanding is that there are plenty of finetuning, system prompt, safety, and other post-training changes that can influence results. (Perhaps unlikely with an SVG generation prompt, but reasonably likely with "reasoning" prompts.) > Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of Isn't that assuming it's answering your questions reasonably well? On a hard benchmark, you may be watching the progression from absolutely miserable to not quite tolerable. You may even be using a constellation of questions to try to delineate the boundary between what it can and can't do.