3 ms·
Downloading this right now, but I have a few problems with it. One is that the books are organized into thousands of zip files, while I'd personally rather the
by bhaney 4y ago
Downloading this right now, but I have a few problems with it.
One is that the books are organized into thousands of zip files, while I'd personally rather they just be organized into directories. I get that most people will benefit from the compression, but I suspect that relying on btrfs's transparent zstd compression will do just fine for me, so I'd rather have a simpler access pattern that doesn't involve extracting or scanning through a zip file whenever I want to look for a book. Not a huge deal. I'll just extract them all I guess, just feels like a waste of IO.
Another thing, why txt? I'd much rather this be a collection of epubs instead, so normal ebook readers can open them with proper formatting, and they'd already be similarly compressed. There'd certainly be some increase in overall size, but I think it'd be worth it.
Another problem is that the index files don't seem to be clean? There are references to books that don't exist, apparently because if whatever automated tool was used to create this couldn't extract the text from a book, the book simply isn't present but it still has an ID and an entry in the index files. Another thing that isn't a huge deal and that I can fix locally, but is a bit of an annoyance.
Once I have this all set up in a way I want, I expect to just be able to serve it with a simple web server and a url rewrite rule instead of the software shipped as part of this project. Nothing against that software, but I'd rather just use a filesystem to organize text files. I guess I'll have plenty of time to think about how best to do it, because the current download ETA is over a week.
- BlackLotus89 4y agohttps://www.offlineos.com/libgen-text.txt https://www.offlineos.com/libgen-text.txt About the missing books fiction and non fiction > if the book is missing (15%), that means the text could not be extracted. > if the book is missing (10%), that means the text could not be extracted. === | why plain text? === - text is smaller (figures are for old .tar.xz, compressed .zip is about double the size) +----------+------------------------------------+------------------------+ | | size | number of files | +----------+----------+-------+-----------------+-----------+------------+ | data set | original | text | compressed text | original | text | +----------+----------+-------+-----------------+-----------+------------+ | lg | 32TB | 0.8TB | 0.32TB | 2,450,000 | 2,220,000 | +----------+----------+-------+-----------------+-----------+------------+ | ff | 2.4TB | 0.5TB | 0.18TB | 2,180,000 | 1,890,000 | +----------+----------+-------+-----------------+-----------+------------+ - text can be viewed on every device. - plain text cannot contain malicious code. pdfs can. - text can be searched and indexed more easily. - why csv? anyone can use csv. it can be read raw in a text editor, opened in excel, or imported into a database. database dumps can be used only by experts.
- bhaney 4y agoI don't see how plain text in zip files is any more searchable than epubs (which are each just plaintext-heavy XML files wrapped in a zip file). And PDFs potentially containing malicious code is an argument against using PDF, not an argument for using plaintext specifically. Being viewable on every device is a decent reason, I just don't personally think it's worth the degradation in reading quality. > if the book is missing (15%), that means the text could not be extracted. I saw that, I just don't understand why a book that isn't included has an entry in the CSV file. If the automated tool fails to extract the text from the book to include in the archive, then why put it in the index file?
- djbusby 4y ago> why put it in the index file? So now you have a list of stuff that should have been included but isn't? Then maybe chase down why and fix?