3 ms·
If you keep the prompt the same at some point the data will appear in training set and we might have answer. So even though today it might be a good check it m
by minraws 1y ago
If you keep the prompt the same at some point the data will appear in training set and we might have answer.
So even though today it might be a good check it might not remain as such a good benchmark.
I think we need a way to keep updating prompts without increasing complexity in someway to properly verify model improvements. ARC Deep Research anyone?
- ljsprague 1y agoWouldn't somebody need to answer the question below? Or do you mean the discussion of its weakness might somehow make it stronger the next time it's trained?
- minraws 1y agoI think it can be both, what happens if discussing weakness provides more relavent links for the question and help the model that is trained scraped web data to learn somehow. I am not sure if the model will need the exact answer or just the backlinks to site where they can find them is enough. Maybe just documenting how to do it could do the job as well...
- red_trumpet 1y agoWell, to test research capabilities, one could just adopt the year (2024->2025) in the prompt.
- minraws 1y agoI am not sure what happens if some site keeps tracking these metrics and that manages to find its way into the training data. There are some NBA fan sites that do keep track of some of these tournament level final metrics.
- qingcharles 1y agoI had o3 "cheat" yesterday. I tried to demo a Deep Research task to a friend, but o3 managed to find the answer immediately in a Reddit comment I'd made after trying out the same problem previously. I was still impressed though!