3 ms·

I'm in agreement LLMs get contaminated with test data, particularly from simonw. What I'm referring to is nobody needs worry about hoarding secret questions fro

I'm in agreement LLMs get contaminated with test data, particularly from simonw. What I'm referring to is nobody needs worry about hoarding secret questions from the public eye to avoid that problem. It is a valid approach... but a bit of a sad path considering. Don't run unpublished private benchmarks or worry about keeping a counted hoard of secret questions. Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of (and the only way it gets there in your lifespan is by reasoning rather than memorization). You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. The important thing here being "rotate questions without concern of having things lined up for it" rather than "fear what happens when you discuss your question".

sfink 2y ago
Ah, I understand your logic now. Fair enough, and since I'm not benchmarking LLMs to begin with, I guess it doesn't much matter whether I fully agree or not. But: > You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. Good point, but it's still somewhat of a concern. Especially if you're benchmarking via a chat interface, my understanding is that there are plenty of finetuning, system prompt, safety, and other post-training changes that can influence results. (Perhaps unlikely with an SVG generation prompt, but reasonably likely with "reasoning" prompts.) > Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of Isn't that assuming it's answering your questions reasonably well? On a hard benchmark, you may be watching the progression from absolutely miserable to not quite tolerable. You may even be using a constellation of questions to try to delineate the boundary between what it can and can't do.

sfink 2y ago

Ah, I understand your logic now. Fair enough, and since I'm not benchmarking LLMs to begin with, I guess it doesn't much matter whether I fully agree or not. But: > You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. Good point, but it's still somewhat of a concern. Especially if you're benchmarking via a chat interface, my understanding is that there are plenty of finetuning, system prompt, safety, and other post-training changes that can influence results. (Perhaps unlikely with an SVG generation prompt, but reasonably likely with "reasoning" prompts.) > Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of Isn't that assuming it's answering your questions reasonably well? On a hard benchmark, you may be watching the progression from absolutely miserable to not quite tolerable. You may even be using a constellation of questions to try to delineate the boundary between what it can and can't do.