3 ms·
If they told you, it would be picked up in a future model's training run.
by kridsdale3 10mo ago
If they told you, it would be picked up in a future model's training run.
- jacobn 10mo agoDon't the models typically train on their input too? I.e. submitting the question also carries a risk/chance of it getting picked up? I guess they get such a large input of queries that they can only realistically check and therefore use a small fraction? Though maybe they've come up with some clever trick to make use of it anyway?
- jerojero 10mo agothey probably dont train on inputs from testing grounds. you dont train on your test data because you need to have that to compare if training is improving or not.
- energy123 10mo agoGiven they asked in on LMArena, yes.
- lambda 10mo agoYeah, probably asking on LMArena makes this an invalid benchmark going forward, especially since I think Google is particular active in testing models on LMArena (as evidenced by the fact that I got their preview for this question). I'll need to find a new one, or actually put together a set of questions to use instead of just a single benchmark.
- nl 10mo agoOpenAI and Anthropic don't train on your questions if you have pressed the opt-out button and are using their UI. LMArena is a different matter.
- _heimdall 10mo agoIs that an issue if you now need a new question to ask?