4 ms·
When the data is secret and unavailable to the company before the test, it doesn’t rely on me trusting the company. When the data is not secret and is availabl
by 542354234235 2y ago
When the data is secret and unavailable to the company before the test, it doesn’t rely on me trusting the company. When the data is not secret and is available to the company, I have to trust that the company did not use that prior knowledge to their advantage. When the company lies and says it did not have access, then later admits that it did have access, is means the data is less trustworthy from my outsider perspective. I don’t think “offense” is a factor at all.
If a scientific paper comes out with “empirical data”, I will still look at the conflicts of interest section. If there are no conflicts of interest listed, but then it is found out that there are multiple conflicts of interest, but the authors promise that while they did not disclose them, they also did not affect the paper, I would be more skeptical. I am not “offended”. I am not “rejecting” the data, but I am taking those factors into account when determining how confident I can be in the validity of the data.
- refulgentis 2y ago> When the company lies and says it did not have access, then later admits that it did have access, is means the data is less trustworthy from my outsider perspective. This isn't what happened? I must be missing something. AFAIK: The FrontierMath people self-reported they had a shared folder the OpenAI people had access to that had a subset of some questions. No one denied anything, no one lied about anything, no one said they didn't have access. There was no data obtained under the table. The motte is "they had data for this one benchmark" The bailey is "they got data under the table"