It is apparently widely suspected that a certain "Books2" dataset mentioned by OpenAI is basically just LibGen: https://blusharkmedia.medium.com/the-ongoing-ba

It is apparently widely suspected that a certain "Books2" dataset mentioned by OpenAI is basically just LibGen: https://blusharkmedia.medium.com/the-ongoing-battle-against-openai-and-the-tumultuous-world-of-ai-book-datasets-b86475e7bf3b https://blusharkmedia.medium.com/the-ongoing-battle-against-... https://techhq.com/2023/09/can-libgen-shadow-library-survive-being-sued-by-textbook-publishers/ https://techhq.com/2023/09/can-libgen-shadow-library-survive... https://www.twitter.com/theshawwn/status/1320282152689336320 https://www.twitter.com/theshawwn/status/1320282152689336320 https://qz.com/openai-books-piracy-microsoft-meta-google-chatgpt-bard-1850757064 https://qz.com/openai-books-piracy-microsoft-meta-google-cha... https://qz.com/shadow-libraries-are-at-the-heart-of-the-mounting-cop-1850621671 https://qz.com/shadow-libraries-are-at-the-heart-of-the-moun... https://goodereader.com/blog/e-book-news/authors-file-lawsuit-against-openai-alleging-using-pirated-content-for-training-chatgpt?doing_wp_cron=1716926858.3886160850524902343750 https://goodereader.com/blog/e-book-news/authors-file-lawsui... When asked about whether this was true, they refused to answer based on confidentiality concerns, then said they had deleted all copies of the dataset, stopped using it, and no longer employed the individuals that compiled it: https://www.businessinsider.com/openai-destroyed-ai-training-datasets-lawsuit-authors-books-copyright-2024-5 https://www.businessinsider.com/openai-destroyed-ai-training... We do know for a fact that the (non-OpenAI-controlled) "Books3" dataset is just "all of bibliotik": https://www.twitter.com/theshawwn/status/1320282149329784833 https://www.twitter.com/theshawwn/status/1320282149329784833 https://github.com/soskek/bookcorpus/issues/27 https://github.com/soskek/bookcorpus/issues/27 And we also apparently know for a fact that this was included in the datasets used to train LLAMA: https://en.wikipedia.org/wiki/The_Pile_(dataset) https://en.wikipedia.org/wiki/The_Pile_(dataset) https://aicopyright.substack.com/p/the-books-used-to-train-llms https://aicopyright.substack.com/p/the-books-used-to-train-l... https://aicopyright.substack.com/p/has-your-book-been-used-to-train https://aicopyright.substack.com/p/has-your-book-been-used-t...