5 ms·

Libgen-text: complete Library Genesis, re-packaged as text only (1.2TB)

145 points by rendx 4y ago

generationP 4y ago
Alas, this isn't very useful for reading the books, except maybe for fiction. However, it should be of great use for indexing and searching in them!
- cpach 4y ago
  Indeed. Quite sweet corpus there!
Qem 4y ago
Does that include sci-hub papers?
pessimizer 4y ago
This is pretty amazing. Somebody was calling for this exact thing in a comment on the next thread over.
userbinator 4y ago
Looking at the details it seems 1.2TB is uncompressed text; but text compresses very well, so it could be much less when compressed.
- bhaney 4y ago
  > it seems 1.2TB is uncompressed text I'm downloading the torrent right now and it's 1.2TB of mostly zip files each containing many txt files. So the 1.2TB does appear to be the compressed size (unless the zip files aren't actually compressed, but that would be weird)
nynx 4y ago
A version of libgen with just epubs would also compress very well.
deleted 4y ago
[deleted]
napier 4y ago
Thanks ever so much! Incredible. Not sure if you saw in the libgen thread next door, but I asked about this earlier today.
keepquestioning 4y ago
Was GPT3 trained on this?
bhaney 4y ago
Downloading this right now, but I have a few problems with it. One is that the books are organized into thousands of zip files, while I'd personally rather they just be organized into directories. I get that most people will benefit from the compression, but I suspect that relying on btrfs's transparent zstd compression will do just fine for me, so I'd rather have a simpler access pattern that doesn't involve extracting or scanning through a zip file whenever I want to look for a book. Not a huge deal. I'll just extract them all I guess, just feels like a waste of IO. Another thing, why txt? I'd much rather this be a collection of epubs instead, so normal ebook readers can open them with proper formatting, and they'd already be similarly compressed. There'd certainly be some increase in overall size, but I think it'd be worth it. Another problem is that the index files don't seem to be clean? There are references to books that don't exist, apparently because if whatever automated tool was used to create this couldn't extract the text from a book, the book simply isn't present but it still has an ID and an entry in the index files. Another thing that isn't a huge deal and that I can fix locally, but is a bit of an annoyance. Once I have this all set up in a way I want, I expect to just be able to serve it with a simple web server and a url rewrite rule instead of the software shipped as part of this project. Nothing against that software, but I'd rather just use a filesystem to organize text files. I guess I'll have plenty of time to think about how best to do it, because the current download ETA is over a week.
- BlackLotus89 4y ago
  https://www.offlineos.com/libgen-text.txt https://www.offlineos.com/libgen-text.txt About the missing books fiction and non fiction > if the book is missing (15%), that means the text could not be extracted. > if the book is missing (10%), that means the text could not be extracted. === | why plain text? === - text is smaller (figures are for old .tar.xz, compressed .zip is about double the size) +----------+------------------------------------+------------------------+ | | size | number of files | +----------+----------+-------+-----------------+-----------+------------+ | data set | original | text | compressed text | original | text | +----------+----------+-------+-----------------+-----------+------------+ | lg | 32TB | 0.8TB | 0.32TB | 2,450,000 | 2,220,000 | +----------+----------+-------+-----------------+-----------+------------+ | ff | 2.4TB | 0.5TB | 0.18TB | 2,180,000 | 1,890,000 | +----------+----------+-------+-----------------+-----------+------------+ - text can be viewed on every device. - plain text cannot contain malicious code. pdfs can. - text can be searched and indexed more easily. - why csv? anyone can use csv. it can be read raw in a text editor, opened in excel, or imported into a database. database dumps can be used only by experts.
ctoth 4y ago
Is this the answer to the question of where to get enough data posed in the article about Chinchilla[0]? If tokenized with the GPT-3 tokenizer, about how many tokens would this be? [0]: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
- lizen_one 4y ago
  I guess the text was extracted using two different methods. One results in 0.8TB and the other in 0.5TB text. 1) I assume 1TB (not TiB) of uncompressed (?) text 2) I assume one character is one byte 3) I assume 5 (actually it seems to be 4.7 in English) characters per word So 1TB/1B/5 = 1.0E12/5 ~= 2.0E11 = 0.2T = 200B words. Your article mentioned that Chinchilla is trained on 1.4T tokens. So there is quite some difference. The article also mentions different mysterious book data sets with 27B tokens, 560B tokens, or 390B tokens. The latter datasets were made by Google. So you are still behind Google massive book dataset even if you use probably the largest book dataset "available" to people or instituions outside of Google. EDIT: I thought I made a mistake, but T stands for trillion or tera which are both 1E12.
yyyk 4y ago
>change from .tar.xz to .zip format. this doubles the size, but speeds up file access from 30s to 0.01s Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.
- VHRanger 4y ago
  Yes, there's a lot better compression available these days, but not everyone has the software installed to unpack them
  - yyyk 4y ago
    We practically need libgen-crystal to use this anyway, so just link zstd to libgen-crystal and now it's available. * Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.
  - unicornporn 4y ago
    > not everyone has the software installed to unpack them People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone". > Yes, there's a lot better compression available these days Tell me more!
    - VHRanger 4y ago
      zstd: https://github.com/facebook/zstd https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
- duskwuff 4y ago
  ZIP is seekable -- you can get a list of files and/or read individual files out of the archive without decompressing the whole thing. TAR+(stream compressor) isn't; the only way to get a list of files or to decompress individual files is to read through the whole thing. The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.
gigatexal 4y ago
4 million books holy smokes
lousken 4y ago
anyone wrote a script to load this into elastic?

145 points by rendx 4y ago

Alas, this isn't very useful for reading the books, except maybe for fiction. However, it should be of great use for indexing and searching in them!

cpach 4y ago
Indeed. Quite sweet corpus there!

Qem 4y ago

Does that include sci-hub papers?

pessimizer 4y ago

This is pretty amazing. Somebody was calling for this exact thing in a comment on the next thread over.

userbinator 4y ago

Looking at the details it seems 1.2TB is uncompressed text; but text compresses very well, so it could be much less when compressed.

bhaney 4y ago
> it seems 1.2TB is uncompressed text I'm downloading the torrent right now and it's 1.2TB of mostly zip files each containing many txt files. So the 1.2TB does appear to be the compressed size (unless the zip files aren't actually compressed, but that would be weird)

nynx 4y ago

A version of libgen with just epubs would also compress very well.

deleted 4y ago

[deleted]

napier 4y ago

Thanks ever so much! Incredible. Not sure if you saw in the libgen thread next door, but I asked about this earlier today.

keepquestioning 4y ago

Was GPT3 trained on this?

bhaney 4y ago

Downloading this right now, but I have a few problems with it. One is that the books are organized into thousands of zip files, while I'd personally rather they just be organized into directories. I get that most people will benefit from the compression, but I suspect that relying on btrfs's transparent zstd compression will do just fine for me, so I'd rather have a simpler access pattern that doesn't involve extracting or scanning through a zip file whenever I want to look for a book. Not a huge deal. I'll just extract them all I guess, just feels like a waste of IO. Another thing, why txt? I'd much rather this be a collection of epubs instead, so normal ebook readers can open them with proper formatting, and they'd already be similarly compressed. There'd certainly be some increase in overall size, but I think it'd be worth it. Another problem is that the index files don't seem to be clean? There are references to books that don't exist, apparently because if whatever automated tool was used to create this couldn't extract the text from a book, the book simply isn't present but it still has an ID and an entry in the index files. Another thing that isn't a huge deal and that I can fix locally, but is a bit of an annoyance. Once I have this all set up in a way I want, I expect to just be able to serve it with a simple web server and a url rewrite rule instead of the software shipped as part of this project. Nothing against that software, but I'd rather just use a filesystem to organize text files. I guess I'll have plenty of time to think about how best to do it, because the current download ETA is over a week.

BlackLotus89 4y ago
https://www.offlineos.com/libgen-text.txt https://www.offlineos.com/libgen-text.txt About the missing books fiction and non fiction > if the book is missing (15%), that means the text could not be extracted. > if the book is missing (10%), that means the text could not be extracted. === | why plain text? === - text is smaller (figures are for old .tar.xz, compressed .zip is about double the size) +----------+------------------------------------+------------------------+ | | size | number of files | +----------+----------+-------+-----------------+-----------+------------+ | data set | original | text | compressed text | original | text | +----------+----------+-------+-----------------+-----------+------------+ | lg | 32TB | 0.8TB | 0.32TB | 2,450,000 | 2,220,000 | +----------+----------+-------+-----------------+-----------+------------+ | ff | 2.4TB | 0.5TB | 0.18TB | 2,180,000 | 1,890,000 | +----------+----------+-------+-----------------+-----------+------------+ - text can be viewed on every device. - plain text cannot contain malicious code. pdfs can. - text can be searched and indexed more easily. - why csv? anyone can use csv. it can be read raw in a text editor, opened in excel, or imported into a database. database dumps can be used only by experts.

ctoth 4y ago

Is this the answer to the question of where to get enough data posed in the article about Chinchilla[0]? If tokenized with the GPT-3 tokenizer, about how many tokens would this be? [0]: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...

lizen_one 4y ago
I guess the text was extracted using two different methods. One results in 0.8TB and the other in 0.5TB text. 1) I assume 1TB (not TiB) of uncompressed (?) text 2) I assume one character is one byte 3) I assume 5 (actually it seems to be 4.7 in English) characters per word So 1TB/1B/5 = 1.0E12/5 ~= 2.0E11 = 0.2T = 200B words. Your article mentioned that Chinchilla is trained on 1.4T tokens. So there is quite some difference. The article also mentions different mysterious book data sets with 27B tokens, 560B tokens, or 390B tokens. The latter datasets were made by Google. So you are still behind Google massive book dataset even if you use probably the largest book dataset "available" to people or instituions outside of Google. EDIT: I thought I made a mistake, but T stands for trillion or tera which are both 1E12.

yyyk 4y ago

>change from .tar.xz to .zip format. this doubles the size, but speeds up file access from 30s to 0.01s Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.

VHRanger 4y ago
Yes, there's a lot better compression available these days, but not everyone has the software installed to unpack them
- yyyk 4y ago
  We practically need libgen-crystal to use this anyway, so just link zstd to libgen-crystal and now it's available. * Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.
- unicornporn 4y ago
  > not everyone has the software installed to unpack them People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone". > Yes, there's a lot better compression available these days Tell me more!
  - VHRanger 4y ago
    zstd: https://github.com/facebook/zstd https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
duskwuff 4y ago
ZIP is seekable -- you can get a list of files and/or read individual files out of the archive without decompressing the whole thing. TAR+(stream compressor) isn't; the only way to get a list of files or to decompress individual files is to read through the whole thing. The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.

gigatexal 4y ago

4 million books holy smokes

lousken 4y ago

anyone wrote a script to load this into elastic?