3 ms·

> We want to emphasize that all the student authors in this paper worked really hard on what could have > been a very interesting and valuable paper had the dat

> We want to emphasize that all the student authors in this paper worked really hard on what could have > been a very interesting and valuable paper had the data been collected with consent. The many > problems with the published work were not the fault of the students. So what did the student authors work "really hard on" if not the same data that was collected without consent? Either all student authors are at fault for working on a paper based on data collected without consent or none are. It's not publishing the paper that is the problem here.

knaik94 3y ago
The issue isn't that the results are necessarily bad from using a dataset with copyrighted exams. Using it for training data is "fine". But a copyrighted eval dataset makes it basically impossible to reproduce the results. One part of the original paper was validating the use of the MITQ dataset as a way to evaluate various models. This is seperate from the issues related to the methodology not agreeing with conclusion that GPT4 could "get an MIT degree". The second conclusion and methodology could have been fixed potentially.
- bjourne 3y ago
  The vast majority of all ml research is using copyrighted data both for training and evaluation.
  - knaik94 3y ago
    I don't mean evaluating a model at the end of training, I mean datasets used to quantify the relative performance across models and algorithms. Copyrighted data would make it harder to share and use, to reproduce results. I don't remember coming across a benchmark dataset that was copyrighted.
    - bjourne 3y ago
      That's very odd because I can't for the life of me think of an ml benchmark dataset that is not copyrighted. Copyrighted datasets are everywhere in ml research.
      - knaik94 3y ago
        Just looking at the eval datasets used in the HF leaderboard, ARC is CC BY-SA, TruthfulQA is Apache, and HellaSwag and MMLU are both MIT. The first dataset many people will explore ml with, MNIST, is GPL-3.0 license. The audio courpus datasets I've used have all been similarly permissively licensed, for example LibriTTS is CC BY 4.0. LJ Speech, one of the main datasets used in Tacotron is public domain. Maybe I should have been more explicit and used the term permissively copyrighted instead to help avoid confusion. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

knaik94 3y ago

The issue isn't that the results are necessarily bad from using a dataset with copyrighted exams. Using it for training data is "fine". But a copyrighted eval dataset makes it basically impossible to reproduce the results. One part of the original paper was validating the use of the MITQ dataset as a way to evaluate various models. This is seperate from the issues related to the methodology not agreeing with conclusion that GPT4 could "get an MIT degree". The second conclusion and methodology could have been fixed potentially.

bjourne 3y ago
The vast majority of all ml research is using copyrighted data both for training and evaluation.
- knaik94 3y ago
  I don't mean evaluating a model at the end of training, I mean datasets used to quantify the relative performance across models and algorithms. Copyrighted data would make it harder to share and use, to reproduce results. I don't remember coming across a benchmark dataset that was copyrighted.
  - bjourne 3y ago
    That's very odd because I can't for the life of me think of an ml benchmark dataset that is not copyrighted. Copyrighted datasets are everywhere in ml research.
    - knaik94 3y ago
      Just looking at the eval datasets used in the HF leaderboard, ARC is CC BY-SA, TruthfulQA is Apache, and HellaSwag and MMLU are both MIT. The first dataset many people will explore ml with, MNIST, is GPL-3.0 license. The audio courpus datasets I've used have all been similarly permissively licensed, for example LibriTTS is CC BY 4.0. LJ Speech, one of the main datasets used in Tacotron is public domain. Maybe I should have been more explicit and used the term permissively copyrighted instead to help avoid confusion. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...