3 ms·
> FRCOphth Part 2 questions were sourced from a textbook for doctors preparing to take the examination [17]. This textbook is not freely available on the intern
by gwern 3y ago
> FRCOphth Part 2 questions were sourced from a textbook for doctors preparing to take the examination [17]. This textbook is not freely available on the internet, making the possibility of its content being included in LLMs’ training datasets unlikely [1].
I can't believe they're serious. They didn't even write any new questions?
- hyperorca 3y agoFunny thing is that this textbook can be easily found on LibGen. I do not know a lot about LLM datasets, but they probably include books from these shadow libraries, right ?
- rdedev 3y agoNot just that but wouldn't the references for the textbook be mostly research papers that would be freely available?
- cameldrv 3y agoNice catch. GPT-3 at least was trained on “Books2” which is very widely suspected of containing all of Libgen and Zlibrary. If the questions are in Libgen, this whole paper is invalid.
- pants2 3y agoConsidering that textbooks are probably the single highest quality source of training data for LLMs, I would be very surprised if OpenAI wasn't buying and scanning textbooks for their own training data (including books that aren't even in Books2).
- kaycey2022 3y agoIt’s highly unlikely that they will spend hundreds of thousands of dollars on buying their own copies when it’s not even remotely clear that doing so will be enough to answer the copyright violation cases they are facing.
- gwern 3y agoI didn't even bother checking Libgen, because if it wasn't in Libgen, it'd be in some other dataset.
- dagmx 3y agoI’m sure they’ll be surprised how many books aren’t freely available on the internet that are in common training sets. It would have taken them a few minutes to learn about the current lawsuits around book piracy. I agree with you. That they didn’t try asking novel questions or even look into what training data is used makes this paper bunk.