4 ms·
The issue isn't that the results are necessarily bad from using a dataset with copyrighted exams. Using it for training data is "fine". But a copyrighted eval d
by knaik94 3y ago
The issue isn't that the results are necessarily bad from using a dataset with copyrighted exams. Using it for training data is "fine". But a copyrighted eval dataset makes it basically impossible to reproduce the results. One part of the original paper was validating the use of the MITQ dataset as a way to evaluate various models.
This is seperate from the issues related to the methodology not agreeing with conclusion that GPT4 could "get an MIT degree".
The second conclusion and methodology could have been fixed potentially.
- bjourne 3y agoThe vast majority of all ml research is using copyrighted data both for training and evaluation.
- knaik94 3y agoI don't mean evaluating a model at the end of training, I mean datasets used to quantify the relative performance across models and algorithms. Copyrighted data would make it harder to share and use, to reproduce results. I don't remember coming across a benchmark dataset that was copyrighted.
- bjourne 3y agoThat's very odd because I can't for the life of me think of an ml benchmark dataset that is not copyrighted. Copyrighted datasets are everywhere in ml research.
- knaik94 3y agoJust looking at the eval datasets used in the HF leaderboard, ARC is CC BY-SA, TruthfulQA is Apache, and HellaSwag and MMLU are both MIT. The first dataset many people will explore ml with, MNIST, is GPL-3.0 license. The audio courpus datasets I've used have all been similarly permissively licensed, for example LibriTTS is CC BY 4.0. LJ Speech, one of the main datasets used in Tacotron is public domain. Maybe I should have been more explicit and used the term permissively copyrighted instead to help avoid confusion. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...