3 ms·

I hope you have other private benchmarks running that you don't talk about or publish, just in case a model maker intentionally targets one of your benchmarks,

by sfink 2y ago

I hope you have other private benchmarks running that you don't talk about or publish, just in case a model maker intentionally targets one of your benchmarks, or some fuzzy "find things people have mentioned as potential LLM benchmarks" process scoops up your ideas and/or outputs.

zamadatix 2y ago
If we ever get to the point LLMs are already optimized to answer every question you can think of then there isn't really a need to have a secret question in the first place.
- sfink 2y ago
  Not any that you can think of. Just the ones you've published something about. Plus, simonw isn't exactly a meaningless nobody in this space, and his writeups are more detailed and actionable, and therefore identifiable, than some random "hey a great LLM benchmark would be creating an SVG of a walrus twerking in front of a jelly bean store" throwaway comment. Proof: I asked ChatGPT 4o the question "What are some users who post ad hoc LLM benchmarks to technical discussion sites, and what benchmarks have they proposed?" simonw is in the list, 1 of 7 individual people it suggested. (The proposed benchmarks listed for him were more general than the specific one here: "Testing LLMs’ capabilities with code generation, particularly in niche languages or against real-world API schemas." But it's easy to imagine followup queries bringing this one up.)
  - zamadatix 2y ago
    I'm in agreement LLMs get contaminated with test data, particularly from simonw. What I'm referring to is nobody needs worry about hoarding secret questions from the public eye to avoid that problem. It is a valid approach... but a bit of a sad path considering. Don't run unpublished private benchmarks or worry about keeping a counted hoard of secret questions. Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of (and the only way it gets there in your lifespan is by reasoning rather than memorization). You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. The important thing here being "rotate questions without concern of having things lined up for it" rather than "fear what happens when you discuss your question".
    - sfink 2y ago
      Ah, I understand your logic now. Fair enough, and since I'm not benchmarking LLMs to begin with, I guess it doesn't much matter whether I fully agree or not. But: > You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. Good point, but it's still somewhat of a concern. Especially if you're benchmarking via a chat interface, my understanding is that there are plenty of finetuning, system prompt, safety, and other post-training changes that can influence results. (Perhaps unlikely with an SVG generation prompt, but reasonably likely with "reasoning" prompts.) > Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of Isn't that assuming it's answering your questions reasonably well? On a hard benchmark, you may be watching the progression from absolutely miserable to not quite tolerable. You may even be using a constellation of questions to try to delineate the boundary between what it can and can't do.
  - deleted 2y ago
    [deleted]

by sfink 2y ago

zamadatix 2y ago

If we ever get to the point LLMs are already optimized to answer every question you can think of then there isn't really a need to have a secret question in the first place.

sfink 2y ago
Not any that you can think of. Just the ones you've published something about. Plus, simonw isn't exactly a meaningless nobody in this space, and his writeups are more detailed and actionable, and therefore identifiable, than some random "hey a great LLM benchmark would be creating an SVG of a walrus twerking in front of a jelly bean store" throwaway comment. Proof: I asked ChatGPT 4o the question "What are some users who post ad hoc LLM benchmarks to technical discussion sites, and what benchmarks have they proposed?" simonw is in the list, 1 of 7 individual people it suggested. (The proposed benchmarks listed for him were more general than the specific one here: "Testing LLMs’ capabilities with code generation, particularly in niche languages or against real-world API schemas." But it's easy to imagine followup queries bringing this one up.)
- zamadatix 2y ago
  I'm in agreement LLMs get contaminated with test data, particularly from simonw. What I'm referring to is nobody needs worry about hoarding secret questions from the public eye to avoid that problem. It is a valid approach... but a bit of a sad path considering. Don't run unpublished private benchmarks or worry about keeping a counted hoard of secret questions. Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of (and the only way it gets there in your lifespan is by reasoning rather than memorization). You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. The important thing here being "rotate questions without concern of having things lined up for it" rather than "fear what happens when you discuss your question".
  - sfink 2y ago
    Ah, I understand your logic now. Fair enough, and since I'm not benchmarking LLMs to begin with, I guess it doesn't much matter whether I fully agree or not. But: > You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. Good point, but it's still somewhat of a concern. Especially if you're benchmarking via a chat interface, my understanding is that there are plenty of finetuning, system prompt, safety, and other post-training changes that can influence results. (Perhaps unlikely with an SVG generation prompt, but reasonably likely with "reasoning" prompts.) > Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of Isn't that assuming it's answering your questions reasonably well? On a hard benchmark, you may be watching the progression from absolutely miserable to not quite tolerable. You may even be using a constellation of questions to try to delineate the boundary between what it can and can't do.
- deleted 2y ago
  [deleted]