4 ms·
Funny thing is that this textbook can be easily found on LibGen. I do not know a lot about LLM datasets, but they probably include books from these shadow libra
by hyperorca 2y ago
Funny thing is that this textbook can be easily found on LibGen. I do not know a lot about LLM datasets, but they probably include books from these shadow libraries, right ?
- rdedev 2y agoNot just that but wouldn't the references for the textbook be mostly research papers that would be freely available?
- cameldrv 2y agoNice catch. GPT-3 at least was trained on “Books2” which is very widely suspected of containing all of Libgen and Zlibrary. If the questions are in Libgen, this whole paper is invalid.
- pants2 2y agoConsidering that textbooks are probably the single highest quality source of training data for LLMs, I would be very surprised if OpenAI wasn't buying and scanning textbooks for their own training data (including books that aren't even in Books2).
- kaycey2022 2y agoIt’s highly unlikely that they will spend hundreds of thousands of dollars on buying their own copies when it’s not even remotely clear that doing so will be enough to answer the copyright violation cases they are facing.
- gwern 2y agoI didn't even bother checking Libgen, because if it wasn't in Libgen, it'd be in some other dataset.