6 ms·
ebooks are a 1-2 mb each max. 81.7 TB are a lot of books, like 42-85 million books.
by tremarley 2y ago
ebooks are a 1-2 mb each max. 81.7 TB are a lot of books, like 42-85 million books.
- thunkingdeep 2y agoI’ve got 70-80mb pirated books, I think because of the illustrations. Guess it depends on the book.
- mateus1 2y agoI don’t think they’re using picture heavy book for LLM training, no?
- moralestapia 2y agoYes they do, there's multimodal models.
- mnsu 2y agoFor multi-modal models, why not? They would be probably some of the best data.
- michaelt 2y agoSometimes the PDF of a book is big because the book's packed with important illustrations and charts - like a textbook or journal paper. Other times a PDF of a book is big because someone scanned it and didn't have trustworthy OCR, so they figured distributing images of text at 1.5 MB per page was better than risking OCR errors.
- WithinReason 2y agoPresumably they didn't create the torrent
- littlestymaar 2y agoEven if they didn't use the illustration(which isn't clear given multimodal models), they'd still make use the text in the books.
- deleted 2y ago[deleted]
- rbanffy 2y agoI don't think they need to be selective. It's not like Meta can run out of storage.
- RIMR 2y agoJust because the LLMs are trained on text doesn't mean that images we're a part of what they downloaded. You clean up the data after you acquire it, not before.
- hulitu 2y agoWhy not ? Do you think that AI doesn't enjoy porn ? /s
- squigz 2y agoIt could be anywhere from a few million to a hundred million https://annas-archive.org/datasets https://annas-archive.org/datasets
- weberer 2y agoThe article says they got datasets from Anna's Archive. It was most likely the scihub/libgen torrent which is 96.0 TB right now and contains 92,872,581 files. That's about 1 megabyte per file. https://annas-archive.org/datasets https://annas-archive.org/datasets
- southernplaces7 2y agoWhere does one find these torrent datasets? Did they download the books in bits and pieces or as a single huge multi-TB file?