6 ms·
Is it possible that some LLM’s are trained on these benchmarks? Which would mean they’re overfitting and are incorrectly ranked? Or am I misunderstanding these
by gitgud 3y ago
Is it possible that some LLM’s are trained on these benchmarks? Which would mean they’re overfitting and are incorrectly ranked? Or am I misunderstanding these benchmarks?…
- bbor 3y agoIt would be a bit of a scandal, and IMO too much hassle to sneak in. These models are trained on massive amounts of text - specifically anticipating which metrics people will care about and generating synthetic data just for them seems extra. But not an expert or OP!
- stu2b50 3y agoI don't think it's a scandal, it's a natural thing that happens when iterating on models. OP doesn't mean they literally train on those tests, but that as a meta-consequence of using those tests as benchmarks, you will adjust the model and hyperparameters in ways that perform better on those tests. For a particular model you try to minimally do this by separating a test and validation set, but on a meta-meta level, it's easy to see it happening.
- jasonfarnon 3y agoYou don't see an engineer at an extremely PR-conscious company at least checking how their model performs on popular benchmarks before rolling it out? And if its performance is lackluster, you do you really see them doing nothing about it? It probably doesn't make a huge difference anyway. I know those old vision models were overfitted to the standard image library benchmarks, but they were still very impressive.
- fbdab103 3y agoFamously, some of the image models were so overtrained they could still yield impressive results if the colors were removed.
- lumost 3y agoThis wasn't so much overtraining, as the models learning something different than what we expected. If you look at a pixel by pixel representation of an image, textures tend to be more significant/unique patterns than shapes. There are some funny studies from the mid 2010s exploring this.
- famouswaffles 3y agoTest leakage is not impossible for some benchmarks. But researchers try to avoid/mitigate that as much as possible for obvious reasons.
- pclmulqdq 3y agoGiven all of the times OpenAI has trained on peoples' examples of "bad" prompts, I am sure they are fine-tuning on these benchmarks. It's the natural thing to do if you are trying to position yourself as the "most accurate" AI.
- famouswaffles 3y agoAssuming they were doing that, Fine-tuning on benchmarks isn't the same as test leakage/testing on training data. No researcher is intentionally training on test data. If it performs about as well in instances it has never seen before (test set) then it's not overfit to the test.
- nightski 3y agoI'm confused, fine-tuning is training. How is that not leakage? I'm hesitant to call them researchers, they are employees of a for-profit company trying to meet investor expectations.
- famouswaffles 3y ago1.You train on the kind of problems you want to solve. you don't report numbers that evaluate performance based on examples it trained on. Datasets will typically have splits, one for training and another for testing. 2. Open ai is capped profit. They are also not a publicly traded company. researchers are researchers regardless of who they work for. Training on test data is especially stupid for commercial applications because customers find that out quick and any reputation is gone.
- deleted 3y ago
- sp332 3y agoYeah, it happens. https://hitz-zentroa.github.io/lm-contamination/blog/ https://hitz-zentroa.github.io/lm-contamination/blog/
- iambateman 3y agoThis is SAT-prep in a nutshell. :)
- option 3y agothat’s why OpenAI didn’t release any details on GPT4 training data blend ;)
- moneywoes 3y agoHow would it even be possible to verify that?
- mdp2021 3y ago"Verify", that's quite a demand; "corroborate", you find queries of the same level which would give satisfactory output upon good performance but fail in a faulty overfitted model.
- stevefan1999 3y agoUnfortunately, Goodhart's law applies on most kind of tests > Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
- FanaHOVA 3y agoPresented with no comment :) https://twitter.com/chhillee/status/1635790330854526981?s=46&t=IVF1sX_TGndxvax1l-hJ0Q https://twitter.com/chhillee/status/1635790330854526981?s=46...
- deleted 3y ago[deleted]
- lumost 3y agoHaving worked on ML products, there is sometimes debate on whether you should train on the test partition prior to prod deployment - after all, why would you ship a worse model to prod? Obviously you can't tell whether the model is better at generalization compared to an alternate technique, and you also incur some overfit risk. But many industrial problems are solvable through memorization.
- sangnoir 3y ago> after all, why would you ship a worse model to prod? ...because you need a control to evaluate how well your product is doing? I know it's a young field, but boy, do some folk love removing the "science" from "data science"
- baobabKoodaa 3y agoYou can evaluate a version of the model that has been trained on one set of data, and ship to production a different model that has been trained on the complete set of data. In many cases one can reasonably infer that the model which has seen all of the data will be better than the model which has seen only some of the data. I'm not claiming that's what happened here, nor am I interested in nitpicking "what counts as 'science'". I'm just saying this is a reasonable thing to do.
- mafuy 3y agoThis is possible if you use e.g. train 1000 models on different subsets of data and verify that each and every one of them is performing well. In that case, you can reasonably infer that another model trained on all data would work well, too. But this is, of course, 1000 times more expensive to do. And if you only train 100, or 10, or 1 model, then the deduction becomes increasingly unstable. So from a practical point of view, it's probably not feasible, because you would put those resources into something else instead that has more ROI.