28 ms·

Pirate Library Mirror: Preserving 7TB of books (that are not in Libgen)

996 points by ValentineC 4y ago

metadat 4y ago
Please don't flag it this time. Folks deserve the option to at least be aware of it and decide for themselves if they wish to pursue it.
- yeetsFromHellL3 4y ago
- deleted 4y ago
  [deleted]
AdmiralAsshat 4y ago
HTTP-only makes me weary of visiting a self-professed piracy site. They couldn't even spring for a Let's Encrypt cert?
- quazar 4y ago
  It's a read only blog with 2 pages. What do you gain for putting this over HTTPS?
  - notriddle 4y ago
    The same thing you always get. Assurance that your free Wi-Fi hotspot isn’t tampering with the page.
    - krick 4y ago
      This, and also MITM (like ISP) needs to make their own request to this site, to know what I read. And, technically, they cannot really be sure it's what I've read, since nothing says that this site is static. I'm not that offended, and torrents are only available via TOR anyway, but I do actually appreciate the sentiment. There's no reason to be not using TLS.
  - ac0lyte 4y ago
    Trust?
    - na85 4y ago
      Malicious sites can still use letsencrypt
  - SquareWheel 4y ago
    Do you really want your ISP to know which piracy sites you frequent? This is all being sent in plain text. Or they could change the content, insert a redirect, or inject ads without your knowledge. TLS is needed on all websites - not just those with interaction.
    - ars 4y ago
      They still know which sites you visit even with https.
      - kenniskrag 4y ago
        only the destination IP. TLS encryption is inside tcp and around the http protocol.
        hombre_fatal 4y ago
        You don’t need to copy and paste your reply everywhere it’s relevant on HN. Even us flea brains can carry your remarks in our head and apply them to similar comments.
        geoffeg 4y ago
        And, unless you setup an appropriate DNS server and the default from your ISP, then they also know that you looked up the site's hostname(s).
      - pessimizer 4y ago
        They don't know the page. In the case of this site it probably doesn't matter, but which page you're looking at is always going to be more interesting and informative than which site you looked at. When the prosecutor is looking through your internet records and they see 50 wikipedia hits in some relevant time period, they're going to be upset that https exists.
    - generalizations 4y ago
      https won't keep your ISP from knowing you visited the site. And the rest of those? For a text-only blog, they seem kinda trivial.
- 1vuio0pswjnm7 4y ago
  https://web.archive.org/web/20220701204054if_/http://pilimi.org/blog-introducing.html https://web.archive.org/web/20220701204054if_/http://pilimi....
- mkl 4y ago
  *wary For some reason I'm seeing this mistake more and more lately. https://en.wiktionary.org/wiki/weary https://en.wiktionary.org/wiki/weary vs https://en.wiktionary.org/wiki/wary https://en.wiktionary.org/wiki/wary
  - InCityDreams 4y ago
    No room for auto-correct?
    - brewdad 4y ago
      Even so, there are enough ESL readers on here, along with native speakers who may not understand the difference, that it makes sense to point it out once in a while. Otherwise we end up in lose/loose situation where I see more people use it wrongly than correctly.
    - mkl 4y ago
      Seems unlikely, as "wary" appears to be the more common word (71M Google results vs 55M). Maybe some keyboard mistakenly does it though.
      - b112 4y ago
        Not sure why the count is relevant here. Both are valid words.
        mkl 4y ago
        But automatically replacing a valid word with a less-common valid word that makes less sense in the context is not something predictive text should do.
        b112 4y ago
        You are presuming it was an automatic replacement. Some still type entire words via a thing called a keyboard, on a computer, others manually pick words to finish, from suggestions, not automatically. Beyond that, google searches are not indicative of current usage, in SMS/messages, emails, etc. Google search spans all webpages, and many text pre-web too.
- superkuh 4y ago
  HTTPS only via a CA (instead of self signing) is even worse for site of questional legality. Then it allows a centralized entity information and control of who can access your site. Yes, you can get a new TLS CA to sign your certs if one gets pressured to kick you out, but that just means the next will too.
- jmprspret 4y ago
  Use Tor to access the site then. There was a post here like a day or so ago about how Tor onion site operators were deanonymised due to their TLS certs for their clearnet mirrors.
uniqueuid 4y ago
It's really funny to think about how the advances of technology keeps changing how we perceive books. 7TB is even a commodity disk these days. And it's a lot less than the torrent of scientific papers that floated around some time ago (that was ~18TB IIRC).
- ars 4y ago
  It's 7TB compressed. If it's text you'd need about 70TB to decompress it. It's probably mostly images though, so probably not quite that bad.
  - emj 4y ago
    I've tried to do lossy compression of epubs with some lines of bash scripts; i.e. removing the images and fonts that were not needed. Many epubs could be downsized to a third of their size, but then I found a book that needed the supplied fonts and gave up. When doing lossy compressions can not have those kind of bugs. What I also found was that many of the images in the epubs were already unuseable and nothing like their counter parts in phsyical books.
    - solarmist 4y ago
      I don’t understand this. Are they epubs of comics or something? Epubs are already compressed (zip).
      - robonerd 4y ago
        It's not terribly uncommon to find an epub with several megabytes of cover art and a few hundred kilobytes of text.
      - deleted 4y ago
        [deleted]
      - kristofferR 4y ago
        Good compression of lots of epub files can likely be way more efficient, as deduplication/compression algorithms can be run on lots of books at the same time. Especially so with a good dictionary.
2OEH8eoCRo0 4y ago
Your movie recommendations suck, ValentineC ;)
ars 4y ago
7TB of compressed text? I don't think humanity has generated that much written words in it's entire existence. Although it would be an interesting Fermi Problem to estimate (and don't forget just how well text compresses). This has to be a lot of duplicates or bad formats (images). This would be far more useful to people with some curating.
- khazhoux 4y ago
  These are most likely PDFs, not plaintext.
- deleted 4y ago
  [deleted]
- Aachen 4y ago
  (Ignoring, as someone else already pointed out, that this is PDFs with images and scans probably, not just plain text... I got curious and did some napkin math about your claim) The top ten countries average north of a hundred thousand books per year each[1], so let's say a million books per year globally because I haven't got the time to extract the numbers and sum them all up exactly. It probably wasn't as much fifty years ago, but we're also completely ignoring the Internet which is way bigger because of user-contributed content (not to mention things like newspapers, meeting notes, etc.), so let's say this was the case for the past fifty years. That's fifty million books. Average book has 85k words[2], and a word is like 5 characters on average. Not all countries write in English, especially some big ones like India and China, so we could probably double it on a global scale but let's go for a conservative 7.5 bytes per word and add a byte for the space (I'm ignoring other punctuation). That comes out to 8.5×85e3×1e6 which is less than a gigabyte and uncompressed. Decent compression iirc makes it a fifth of the size, so 145MB compressed. I've still got to be an order of magnitude off because we write a whole lot more than just books (and even if we were just looking at books: there are also book revisions, drafts, etc. I'm just counting the published words), but that's still less than expected. In conclusion, it comes out to about 1/50'000th of 7TB compressed, which I would say lends some credence to the claim—even if I feel like I must have made a mistake somewhere because 145MB for ~all books of the past 50 years from all countries seems quite little. [1] https://en.wikipedia.org/wiki/Books_published_per_country_per_year https://en.wikipedia.org/wiki/Books_published_per_country_pe... [2] a few sources on ddg, e.g. https://www.tckpublishing.com/how-long-should-a-book-be/ https://www.tckpublishing.com/how-long-should-a-book-be/
generalizations 4y ago
I wonder if there are any search engines dedicated to indexing these kinds of libraries. I know there's a decent one just for scihub, but it would be awesome if I could do a Google-style search that returned the contents of books, magazines and journal articles instead of just websites.
- delusional 4y ago
  Wasn't that what google books was supposed to be?
  - voisin 4y ago
    I assume Google abandoned this along with all their earlier mission statements in favour of building another chat app.
    - emanuensis 4y ago
      Supposedly all at Library of Congress (LoC) anyone have experience using it there?
    - b112 4y ago
      I can't imagine a life at google. So much promise, all turned to ash.
      - UkrainianJew 4y ago
        More like an ad company driven by dark patterns that needs to pretend being good and nice and innovative as otherwise it will lose goodwill just like Facebook did.
        klipt 4y ago
        I think the Google founders were inherently nicer people than the Facebook founder, but their companies ended up converging on advertising because that's really the only way their products could be profitable.
Larrikin 4y ago
Is there an index of what's included. 7 tb is alot to ask for simply upholding an ideal
- robonerd 4y ago
  It apparently comes with a index in the form of a MySQL database that contains title, author, description, and filetype.
robonerd 4y ago
> We will release the data in stages, as we are still processing the files. Right now the metadata file and a few of the torrents are available. Note that the torrent files are only available through our TOR mirror. Presently, only the first four of several dozens of parts are available.
dkjaudyeqooe 4y ago
To be fair, Z-Library doesn't charge unless you want to download more than 10 books per 24 hour period. That's per account and although they ask you not to open multiple accounts they don't seem to do anything to stop you.
- PostOnce 4y ago
  What kind of fairness can there be in charging for stolen books? I believe in free access to education, but charging for these books they have no rights to is a whole other thing.
  - Sparkle-san 4y ago
    It would depend on how they use the funds. I wouldn't be surprised if bandwidth expenses made up a majority of what it cost to run Z-Library and that money has to come from somewhere.
  - mdp2021 4y ago
    Running costs.
  - fancyfredbot 4y ago
    You don't have to pay them if you go steal them yourself. If you find it more convenient to pay another thief to do it for you then I don't think that's significantly less fair.
    - UmbertoNoEco 4y ago
      All those India/Malasya/Guatemala kids stealing a 200 USD PDE book from Pearson should go to jail... in America... Assange style
      - dang 4y ago
        Could you please top posting unsubstantive comments to HN? You've been doing it a lot, unfortunately, and we're trying for a different sort of discussion here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
krick 4y ago
So, if I get it right. First there was Libgen, which is mirrorable. Then, some Z-Library copied Libgen and added some more books, without making it mirrorable. The goal is to make these new books, which are not mirrorable — mirrorable (i.e. to "preserve" them). So, why not just re-upload them to Libgen, then? I guess somebody will do that now anyway, but you could easily done it in the first place, without making your own mirror, which is not a mirror of Libgen. Just upload them to Libgen and make a mirror of Libgen.
- facethewolf 4y ago
  From their FAQ: > Q: Should the Z-Library collection be added to Library Genesis? > A: Yes! However, it is tricky. Library Genesis splits out its collection between non-fiction and fiction. They also have relatively high quality standards. If you are interested in organizing all the books to meet their requirements, let us know.
  - krick 4y ago
    Oh, it didn't come up to me to navigate "backwards". Thanks. Actually, their whole FAQ is quite more enlightening than just the linked page. http://pilimi.org/faq.html http://pilimi.org/faq.html
    - pnutjam 4y ago
      I found a book I've been looking for, but it was only an image scan pdf. I OCR'ed it, and I'm slowly fixing the errors and converting to epub. Tedious, but interesting.
      - sudosysgen 4y ago
        Kudos to you, but I would probably be wary of admitting to participate in the illegal book piracy scene unless your pseudonymity is really tight. Thanks a lot for helping out, though.
        4y ago
sacrosanct 4y ago
This really needs to be hosted on a Tor hidden service. Clearnet sites are easy to take down.
- progman32 4y ago
  The homepage has a link to the onion site.
  - sacrosanct 4y ago
    So why does it have a clearnet address? To have more reach? What’s their threat model such that a clearnet presence could possibly out the people behind this?
    - progman32 4y ago
      I don't know. I'd guess reach. Probably wouldn't be on HN if it were darknet only.
    - ipaddr 4y ago
      There is no ssl so no cert fingerprint shodan matching leak
      - 0xbkt 4y ago
        Mind elaborating?
        UncleEntity 4y ago
        “They” deanonymize anonymous site operators through cert reuse because people get lazy/stupid. Probably 100% doesn’t matter in this case.
- fsflover 4y ago
  Or, better on torrents inside I2P: http://geti2p.net http://geti2p.net.
moritonal 4y ago
It's a shame that this would have been a textbook case for using IPFS and yet that wasn't the default. Books are naturally immutable, and could be structured into sub-categories whilst enjoying the benefits of deduplication.
- UkrainianJew 4y ago
  That would paint a huge red target mark on filecoin's back, given that they raised over a quarter-billion. They would take it down faster than you can have an 8TB HDD shipped to you.
  - betwixthewires 4y ago
    They can't take it down.
    - jmprspret 4y ago
      so you're telling me there's CSAM on IPFS that feds "can't" take down? Somehow I doubt that.
      - betwixthewires 4y ago
        Somehow? How? Do you know how IPFS works? They can't take down torrents either. But we were talking about filecoin raking something down from ipfs. They can't do it.
    - UkrainianJew 4y ago
      They can and will use legal means to go after anyone trying to use the network this way in order to make the content less discoverable and discourage others from following suit. Being coined as the Napster on Blockchain is the worst possible PR one can think of in their situation.
      - trasz 4y ago
        I suspect Russia could now help with this one :-)
hour_glass 4y ago
What kind of books are on here? I can find what im looking for ~90% of the time on libgen
stavros 4y ago
I see things like this, and I wonder why the following software doesn't exist: I want a piece of software to which I can add a collection of files, say multiple TB. The software will then behave a bit like a BitTorrent tracker, and know which peer has which files. A peer joining this swarm will be able to say "I want to donate X GB of space", and the tracker would tell it "OK, then download and seed these files, which are the least seeded". The peer would download the files from the rest of the swarm and make them available to it. Then, a request layer on top of the swarm could be used to request a file from the peer which had it. Adding/removing files to this collection would also need to be a feature. Does anyone know if anything like this exists? If not, how easy would it be to make something like it out of BitTorrent? I might give it a go.
- er4hn 4y ago
  This sounds very similar to ipfs. How does it differ?
  - stavros 4y ago
    IPFS can't give you the least-seeded files. It can't give you any files, you have to manually pin the set you want, and you pin all of it, and it won't automatically change. It's not very convenient for archival, whereas the system I'm talking about would be (in my opinion, anyway).
    - paskozdilar 4y ago
      (How) would you filter junk?
      - stavros 4y ago
        You'd trust the publisher, so you'd say "I want to help the Internet Archive with its archiving" and they'd be responsible for what files they pushed to your storage. This is basically an opportunistic, distributed filesystem that's designed to work with high latency links. Only the tracker can write to its nodes, but anyone can read.
jrm4 4y ago
Again, it's just insane to me that we don't even much have a meaningful discussion of: "Hey, wait, literally everyone could have the entire library of Alexandria in their house for a couple hundred bucks per person. Like, all the knowledge ever. Maybe that should be considered the good default of things. At least one in every town that everyone could use, for free, forever, without restriction to ANY of the knowledge anyone desires."
- gmiller123456 4y ago
  As a kid, my parents had a full copy of the Library of Alexandria on the coffee table. Everyone else just called it an ashtray. (Sorry, too soon?)
  - throwaway742 4y ago
    I haven't actually laughed for like a month. Thanks.
  - Sakos 4y ago
    :'(
  - waplot 4y ago
    have your upvote. now show yourself out.
  - ta988 4y ago
    It likely didn't burn and was mostly destroyed by purging of scholars and abandon. https://en.m.wikipedia.org/wiki/Library_of_Alexandria https://en.m.wikipedia.org/wiki/Library_of_Alexandria
- mylons 4y ago
  what's even more mind boggling is that library is more or less publicly accessible _now_ via a home internet connection or a connection from your... friendly neighborhood library! but you're right, having unmitigated access to that is even more incredible.
- malloryerik 4y ago
tanepiper 4y ago
Reminds me of that Twilight Zone episode - all the books in the world and no time to read them.
deleted 4y ago
[deleted]
Dorothy014 4y ago
[dead]
plmu 4y ago
Excellent. I hope more and more people start to see how absurd and evil the concept of "intellectual property" is. It should be totally rejected and any form of keeping useful information for yourself should be shunned and tabooized. In todays world, many have been programmed to believe that the would could not exits without such immoral restrictions, which is horrible.
- jesterson 4y ago
  There should be a fine line in intellectual property rights. I see where you are coming from - quite often intellectual property is used a a moat to protect insane revenues and, as a repercussion, delay or slowdown our progress as humanity. But it is also use to protect unique creator revenue and encourage to create more. If you ask where the fine line should be I have no immediate answer, but abolishing intellectual property rights just like enforcing them at all costs doesn't seem to be the optimal course of action to me.
  - js8 4y ago
    I think that's pretty much the case for shortening time frame of IP rights. Which means, they shouldn't be treated as property has been traditionally treated. Although as a socialist, I think perhaps we shouldn't have time unlimited property rights (above certain reasonable boundary, say $10M) in general.
    - jesterson 4y ago
      Interesting, but being very opposite of socialist myself I am wholeheartedly with you here on limiting timeframe of IP. It partially solves the problem. And this arguably should be extended to tangible assets as well - I like Singapore model where housing property is sold for specific timeframe. It simplifies a lot of redevelopment.
      - blangk 4y ago
        Something like the homestead principle?
752963e64 4y ago
Inflated head goes deflate subsequently :D You don't need book to be smart... you need a brain :)
robertwt7 4y ago
I would love to contribute on seeding the torrents. However what happens when there are new books in z-lib? Is this only 1 time mirror? How do we expand the collection after a certain amount of time.
- Tepix 4y ago
  Apparently the books are numbered and they will be adding new torrents regularly.
deknos 4y ago
is there actually a command line tool which can bulkdownload according to filters from libgen/mirrors? i do not care about speed, rate limiting would be totally okay, but i do not see such tools, which iteratively go over each mirror and check for content in a service friendly way
flanfly 4y ago
One of the projects on my secret TODO list is feeding Libgen into Elastic Search to get a cross referenced full text search. For now the hardware is prohibitively expensive, but time is on my side. I'm sure redundantly indexing a few 10s of TB will become trivial before the end of the decade.
- sitkack 4y ago
  I suggest you just do it. You can generate the indexes on some cloud VMs and save them. It would cost under 100$.
GTP 4y ago
If they are looking for seeders they should break the 7TB collection in multiple torrents, as I think most people don't have 7TB to spare
- generalizations 4y ago
  If it's not a compressed archive, you can always just download part of it.
- Tepix 4y ago
  That appears to be what they are doing
_joel 4y ago
Is there a set of magent links? Seems a bit OTT having to log in to tor to get the torrents when they could be DHT.
cookiengineer 4y ago
This project kind of reminds me of "the eye" [1] which had a _huge_ collection of computer science and hacking related books. Somewhere around last year they had datacenters burning down due to a natural disaster, but I'm hoping they can recover. It's an amazing project. [1] https://the-eye.eu/ https://the-eye.eu/
stiray 4y ago
Hm, from some site on the internet, this are the stats: Enabled users: 5629 Active today: 986 Active this week: 2542 Active this month: 4228 Torrents: 566598 Total Size: 23.40 TiB Retail Torrents: 421313 Creators: 401395 Seeders: 2421540 Leechers: 232 Snatches: 16523563 Transferred: 398.22 TiB And those are only books. Library of Alexandria is already here. The only problem is that there is so many books and so little time :( , which might be a far bigger problem than book accessibility. To find the time to read them. There is at least 1/8 of books that I would love to read. But I don't have the time to actually pull it off, even if I am not doing any social networking etc., but the amount is really huge. Maybe, for a startup, someone could index all the content, not only share it.
pilimi_anna 4y ago
For those following along, we just added the full set of torrents to our TOR website.

996 points by ValentineC 4y ago

Please don't flag it this time. Folks deserve the option to at least be aware of it and decide for themselves if they wish to pursue it.

yeetsFromHellL3 4y ago
deleted 4y ago
[deleted]

AdmiralAsshat 4y ago

HTTP-only makes me weary of visiting a self-professed piracy site. They couldn't even spring for a Let's Encrypt cert?

quazar 4y ago
It's a read only blog with 2 pages. What do you gain for putting this over HTTPS?
- notriddle 4y ago
  The same thing you always get. Assurance that your free Wi-Fi hotspot isn’t tampering with the page.
  - krick 4y ago
    This, and also MITM (like ISP) needs to make their own request to this site, to know what I read. And, technically, they cannot really be sure it's what I've read, since nothing says that this site is static. I'm not that offended, and torrents are only available via TOR anyway, but I do actually appreciate the sentiment. There's no reason to be not using TLS.
- ac0lyte 4y ago
  Trust?
  - na85 4y ago
    Malicious sites can still use letsencrypt
- SquareWheel 4y ago
  Do you really want your ISP to know which piracy sites you frequent? This is all being sent in plain text. Or they could change the content, insert a redirect, or inject ads without your knowledge. TLS is needed on all websites - not just those with interaction.
  - ars 4y ago
    They still know which sites you visit even with https.
    - kenniskrag 4y ago
      only the destination IP. TLS encryption is inside tcp and around the http protocol.
      - hombre_fatal 4y ago
        You don’t need to copy and paste your reply everywhere it’s relevant on HN. Even us flea brains can carry your remarks in our head and apply them to similar comments.
      - geoffeg 4y ago
        And, unless you setup an appropriate DNS server and the default from your ISP, then they also know that you looked up the site's hostname(s).
    - pessimizer 4y ago
      They don't know the page. In the case of this site it probably doesn't matter, but which page you're looking at is always going to be more interesting and informative than which site you looked at. When the prosecutor is looking through your internet records and they see 50 wikipedia hits in some relevant time period, they're going to be upset that https exists.
  - generalizations 4y ago
    https won't keep your ISP from knowing you visited the site. And the rest of those? For a text-only blog, they seem kinda trivial.
1vuio0pswjnm7 4y ago
https://web.archive.org/web/20220701204054if_/http://pilimi.org/blog-introducing.html https://web.archive.org/web/20220701204054if_/http://pilimi....
mkl 4y ago
*wary For some reason I'm seeing this mistake more and more lately. https://en.wiktionary.org/wiki/weary https://en.wiktionary.org/wiki/weary vs https://en.wiktionary.org/wiki/wary https://en.wiktionary.org/wiki/wary
- InCityDreams 4y ago
  No room for auto-correct?
  - brewdad 4y ago
    Even so, there are enough ESL readers on here, along with native speakers who may not understand the difference, that it makes sense to point it out once in a while. Otherwise we end up in lose/loose situation where I see more people use it wrongly than correctly.
  - mkl 4y ago
    Seems unlikely, as "wary" appears to be the more common word (71M Google results vs 55M). Maybe some keyboard mistakenly does it though.
    - b112 4y ago
      Not sure why the count is relevant here. Both are valid words.
      - mkl 4y ago
        But automatically replacing a valid word with a less-common valid word that makes less sense in the context is not something predictive text should do.
        b112 4y ago
        You are presuming it was an automatic replacement. Some still type entire words via a thing called a keyboard, on a computer, others manually pick words to finish, from suggestions, not automatically. Beyond that, google searches are not indicative of current usage, in SMS/messages, emails, etc. Google search spans all webpages, and many text pre-web too.
superkuh 4y ago
HTTPS only via a CA (instead of self signing) is even worse for site of questional legality. Then it allows a centralized entity information and control of who can access your site. Yes, you can get a new TLS CA to sign your certs if one gets pressured to kick you out, but that just means the next will too.
jmprspret 4y ago
Use Tor to access the site then. There was a post here like a day or so ago about how Tor onion site operators were deanonymised due to their TLS certs for their clearnet mirrors.

uniqueuid 4y ago

It's really funny to think about how the advances of technology keeps changing how we perceive books. 7TB is even a commodity disk these days. And it's a lot less than the torrent of scientific papers that floated around some time ago (that was ~18TB IIRC).

ars 4y ago
It's 7TB compressed. If it's text you'd need about 70TB to decompress it. It's probably mostly images though, so probably not quite that bad.
- emj 4y ago
  I've tried to do lossy compression of epubs with some lines of bash scripts; i.e. removing the images and fonts that were not needed. Many epubs could be downsized to a third of their size, but then I found a book that needed the supplied fonts and gave up. When doing lossy compressions can not have those kind of bugs. What I also found was that many of the images in the epubs were already unuseable and nothing like their counter parts in phsyical books.
  - solarmist 4y ago
    I don’t understand this. Are they epubs of comics or something? Epubs are already compressed (zip).
    - robonerd 4y ago
      It's not terribly uncommon to find an epub with several megabytes of cover art and a few hundred kilobytes of text.
    - deleted 4y ago
      [deleted]
    - kristofferR 4y ago
      Good compression of lots of epub files can likely be way more efficient, as deduplication/compression algorithms can be run on lots of books at the same time. Especially so with a good dictionary.

2OEH8eoCRo0 4y ago

Your movie recommendations suck, ValentineC ;)

ars 4y ago

7TB of compressed text? I don't think humanity has generated that much written words in it's entire existence. Although it would be an interesting Fermi Problem to estimate (and don't forget just how well text compresses). This has to be a lot of duplicates or bad formats (images). This would be far more useful to people with some curating.

khazhoux 4y ago
These are most likely PDFs, not plaintext.
deleted 4y ago
[deleted]
Aachen 4y ago
(Ignoring, as someone else already pointed out, that this is PDFs with images and scans probably, not just plain text... I got curious and did some napkin math about your claim) The top ten countries average north of a hundred thousand books per year each[1], so let's say a million books per year globally because I haven't got the time to extract the numbers and sum them all up exactly. It probably wasn't as much fifty years ago, but we're also completely ignoring the Internet which is way bigger because of user-contributed content (not to mention things like newspapers, meeting notes, etc.), so let's say this was the case for the past fifty years. That's fifty million books. Average book has 85k words[2], and a word is like 5 characters on average. Not all countries write in English, especially some big ones like India and China, so we could probably double it on a global scale but let's go for a conservative 7.5 bytes per word and add a byte for the space (I'm ignoring other punctuation). That comes out to 8.5×85e3×1e6 which is less than a gigabyte and uncompressed. Decent compression iirc makes it a fifth of the size, so 145MB compressed. I've still got to be an order of magnitude off because we write a whole lot more than just books (and even if we were just looking at books: there are also book revisions, drafts, etc. I'm just counting the published words), but that's still less than expected. In conclusion, it comes out to about 1/50'000th of 7TB compressed, which I would say lends some credence to the claim—even if I feel like I must have made a mistake somewhere because 145MB for ~all books of the past 50 years from all countries seems quite little. [1] https://en.wikipedia.org/wiki/Books_published_per_country_per_year https://en.wikipedia.org/wiki/Books_published_per_country_pe... [2] a few sources on ddg, e.g. https://www.tckpublishing.com/how-long-should-a-book-be/ https://www.tckpublishing.com/how-long-should-a-book-be/

generalizations 4y ago

I wonder if there are any search engines dedicated to indexing these kinds of libraries. I know there's a decent one just for scihub, but it would be awesome if I could do a Google-style search that returned the contents of books, magazines and journal articles instead of just websites.

delusional 4y ago
Wasn't that what google books was supposed to be?
- voisin 4y ago
  I assume Google abandoned this along with all their earlier mission statements in favour of building another chat app.
  - emanuensis 4y ago
    Supposedly all at Library of Congress (LoC) anyone have experience using it there?
  - b112 4y ago
    I can't imagine a life at google. So much promise, all turned to ash.
    - UkrainianJew 4y ago
      More like an ad company driven by dark patterns that needs to pretend being good and nice and innovative as otherwise it will lose goodwill just like Facebook did.
      - klipt 4y ago
        I think the Google founders were inherently nicer people than the Facebook founder, but their companies ended up converging on advertising because that's really the only way their products could be profitable.

Larrikin 4y ago

Is there an index of what's included. 7 tb is alot to ask for simply upholding an ideal

robonerd 4y ago
It apparently comes with a index in the form of a MySQL database that contains title, author, description, and filetype.

robonerd 4y ago

> We will release the data in stages, as we are still processing the files. Right now the metadata file and a few of the torrents are available. Note that the torrent files are only available through our TOR mirror. Presently, only the first four of several dozens of parts are available.

dkjaudyeqooe 4y ago

To be fair, Z-Library doesn't charge unless you want to download more than 10 books per 24 hour period. That's per account and although they ask you not to open multiple accounts they don't seem to do anything to stop you.

PostOnce 4y ago
What kind of fairness can there be in charging for stolen books? I believe in free access to education, but charging for these books they have no rights to is a whole other thing.
- Sparkle-san 4y ago
  It would depend on how they use the funds. I wouldn't be surprised if bandwidth expenses made up a majority of what it cost to run Z-Library and that money has to come from somewhere.
- mdp2021 4y ago
  Running costs.
- fancyfredbot 4y ago
  You don't have to pay them if you go steal them yourself. If you find it more convenient to pay another thief to do it for you then I don't think that's significantly less fair.
  - UmbertoNoEco 4y ago
    All those India/Malasya/Guatemala kids stealing a 200 USD PDE book from Pearson should go to jail... in America... Assange style
    - dang 4y ago
      Could you please top posting unsubstantive comments to HN? You've been doing it a lot, unfortunately, and we're trying for a different sort of discussion here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

krick 4y ago

So, if I get it right. First there was Libgen, which is mirrorable. Then, some Z-Library copied Libgen and added some more books, without making it mirrorable. The goal is to make these new books, which are not mirrorable — mirrorable (i.e. to "preserve" them). So, why not just re-upload them to Libgen, then? I guess somebody will do that now anyway, but you could easily done it in the first place, without making your own mirror, which is not a mirror of Libgen. Just upload them to Libgen and make a mirror of Libgen.

facethewolf 4y ago
From their FAQ: > Q: Should the Z-Library collection be added to Library Genesis? > A: Yes! However, it is tricky. Library Genesis splits out its collection between non-fiction and fiction. They also have relatively high quality standards. If you are interested in organizing all the books to meet their requirements, let us know.
- krick 4y ago
  Oh, it didn't come up to me to navigate "backwards". Thanks. Actually, their whole FAQ is quite more enlightening than just the linked page. http://pilimi.org/faq.html http://pilimi.org/faq.html
  - pnutjam 4y ago
    I found a book I've been looking for, but it was only an image scan pdf. I OCR'ed it, and I'm slowly fixing the errors and converting to epub. Tedious, but interesting.
    - sudosysgen 4y ago
      Kudos to you, but I would probably be wary of admitting to participate in the illegal book piracy scene unless your pseudonymity is really tight. Thanks a lot for helping out, though.
      - 4y ago

sacrosanct 4y ago

This really needs to be hosted on a Tor hidden service. Clearnet sites are easy to take down.

progman32 4y ago
The homepage has a link to the onion site.
- sacrosanct 4y ago
  So why does it have a clearnet address? To have more reach? What’s their threat model such that a clearnet presence could possibly out the people behind this?
  - progman32 4y ago
    I don't know. I'd guess reach. Probably wouldn't be on HN if it were darknet only.
  - ipaddr 4y ago
    There is no ssl so no cert fingerprint shodan matching leak
    - 0xbkt 4y ago
      Mind elaborating?
      - UncleEntity 4y ago
        “They” deanonymize anonymous site operators through cert reuse because people get lazy/stupid. Probably 100% doesn’t matter in this case.
fsflover 4y ago
Or, better on torrents inside I2P: http://geti2p.net http://geti2p.net.

moritonal 4y ago

It's a shame that this would have been a textbook case for using IPFS and yet that wasn't the default. Books are naturally immutable, and could be structured into sub-categories whilst enjoying the benefits of deduplication.

UkrainianJew 4y ago
That would paint a huge red target mark on filecoin's back, given that they raised over a quarter-billion. They would take it down faster than you can have an 8TB HDD shipped to you.
- betwixthewires 4y ago
  They can't take it down.
  - jmprspret 4y ago
    so you're telling me there's CSAM on IPFS that feds "can't" take down? Somehow I doubt that.
    - betwixthewires 4y ago
      Somehow? How? Do you know how IPFS works? They can't take down torrents either. But we were talking about filecoin raking something down from ipfs. They can't do it.
  - UkrainianJew 4y ago
    They can and will use legal means to go after anyone trying to use the network this way in order to make the content less discoverable and discourage others from following suit. Being coined as the Napster on Blockchain is the worst possible PR one can think of in their situation.
    - trasz 4y ago
      I suspect Russia could now help with this one :-)

hour_glass 4y ago

What kind of books are on here? I can find what im looking for ~90% of the time on libgen

stavros 4y ago

I see things like this, and I wonder why the following software doesn't exist: I want a piece of software to which I can add a collection of files, say multiple TB. The software will then behave a bit like a BitTorrent tracker, and know which peer has which files. A peer joining this swarm will be able to say "I want to donate X GB of space", and the tracker would tell it "OK, then download and seed these files, which are the least seeded". The peer would download the files from the rest of the swarm and make them available to it. Then, a request layer on top of the swarm could be used to request a file from the peer which had it. Adding/removing files to this collection would also need to be a feature. Does anyone know if anything like this exists? If not, how easy would it be to make something like it out of BitTorrent? I might give it a go.

er4hn 4y ago
This sounds very similar to ipfs. How does it differ?
- stavros 4y ago
  IPFS can't give you the least-seeded files. It can't give you any files, you have to manually pin the set you want, and you pin all of it, and it won't automatically change. It's not very convenient for archival, whereas the system I'm talking about would be (in my opinion, anyway).
  - paskozdilar 4y ago
    (How) would you filter junk?
    - stavros 4y ago
      You'd trust the publisher, so you'd say "I want to help the Internet Archive with its archiving" and they'd be responsible for what files they pushed to your storage. This is basically an opportunistic, distributed filesystem that's designed to work with high latency links. Only the tracker can write to its nodes, but anyone can read.

jrm4 4y ago

Again, it's just insane to me that we don't even much have a meaningful discussion of: "Hey, wait, literally everyone could have the entire library of Alexandria in their house for a couple hundred bucks per person. Like, all the knowledge ever. Maybe that should be considered the good default of things. At least one in every town that everyone could use, for free, forever, without restriction to ANY of the knowledge anyone desires."

gmiller123456 4y ago
As a kid, my parents had a full copy of the Library of Alexandria on the coffee table. Everyone else just called it an ashtray. (Sorry, too soon?)
- throwaway742 4y ago
  I haven't actually laughed for like a month. Thanks.
- Sakos 4y ago
  :'(
- waplot 4y ago
  have your upvote. now show yourself out.
- ta988 4y ago
  It likely didn't burn and was mostly destroyed by purging of scholars and abandon. https://en.m.wikipedia.org/wiki/Library_of_Alexandria https://en.m.wikipedia.org/wiki/Library_of_Alexandria
mylons 4y ago
what's even more mind boggling is that library is more or less publicly accessible _now_ via a home internet connection or a connection from your... friendly neighborhood library! but you're right, having unmitigated access to that is even more incredible.
malloryerik 4y ago

tanepiper 4y ago

Reminds me of that Twilight Zone episode - all the books in the world and no time to read them.

deleted 4y ago

[deleted]

Dorothy014 4y ago

[dead]

plmu 4y ago

Excellent. I hope more and more people start to see how absurd and evil the concept of "intellectual property" is. It should be totally rejected and any form of keeping useful information for yourself should be shunned and tabooized. In todays world, many have been programmed to believe that the would could not exits without such immoral restrictions, which is horrible.

jesterson 4y ago
There should be a fine line in intellectual property rights. I see where you are coming from - quite often intellectual property is used a a moat to protect insane revenues and, as a repercussion, delay or slowdown our progress as humanity. But it is also use to protect unique creator revenue and encourage to create more. If you ask where the fine line should be I have no immediate answer, but abolishing intellectual property rights just like enforcing them at all costs doesn't seem to be the optimal course of action to me.
- js8 4y ago
  I think that's pretty much the case for shortening time frame of IP rights. Which means, they shouldn't be treated as property has been traditionally treated. Although as a socialist, I think perhaps we shouldn't have time unlimited property rights (above certain reasonable boundary, say $10M) in general.
  - jesterson 4y ago
    Interesting, but being very opposite of socialist myself I am wholeheartedly with you here on limiting timeframe of IP. It partially solves the problem. And this arguably should be extended to tangible assets as well - I like Singapore model where housing property is sold for specific timeframe. It simplifies a lot of redevelopment.
    - blangk 4y ago
      Something like the homestead principle?

752963e64 4y ago

Inflated head goes deflate subsequently :D You don't need book to be smart... you need a brain :)

robertwt7 4y ago

I would love to contribute on seeding the torrents. However what happens when there are new books in z-lib? Is this only 1 time mirror? How do we expand the collection after a certain amount of time.

Tepix 4y ago
Apparently the books are numbered and they will be adding new torrents regularly.

deknos 4y ago

is there actually a command line tool which can bulkdownload according to filters from libgen/mirrors? i do not care about speed, rate limiting would be totally okay, but i do not see such tools, which iteratively go over each mirror and check for content in a service friendly way

flanfly 4y ago

One of the projects on my secret TODO list is feeding Libgen into Elastic Search to get a cross referenced full text search. For now the hardware is prohibitively expensive, but time is on my side. I'm sure redundantly indexing a few 10s of TB will become trivial before the end of the decade.

sitkack 4y ago
I suggest you just do it. You can generate the indexes on some cloud VMs and save them. It would cost under 100$.

GTP 4y ago

If they are looking for seeders they should break the 7TB collection in multiple torrents, as I think most people don't have 7TB to spare

generalizations 4y ago
If it's not a compressed archive, you can always just download part of it.
Tepix 4y ago
That appears to be what they are doing

_joel 4y ago

Is there a set of magent links? Seems a bit OTT having to log in to tor to get the torrents when they could be DHT.

cookiengineer 4y ago

This project kind of reminds me of "the eye" [1] which had a _huge_ collection of computer science and hacking related books. Somewhere around last year they had datacenters burning down due to a natural disaster, but I'm hoping they can recover. It's an amazing project. [1] https://the-eye.eu/ https://the-eye.eu/

stiray 4y ago

Hm, from some site on the internet, this are the stats: Enabled users: 5629 Active today: 986 Active this week: 2542 Active this month: 4228 Torrents: 566598 Total Size: 23.40 TiB Retail Torrents: 421313 Creators: 401395 Seeders: 2421540 Leechers: 232 Snatches: 16523563 Transferred: 398.22 TiB And those are only books. Library of Alexandria is already here. The only problem is that there is so many books and so little time :( , which might be a far bigger problem than book accessibility. To find the time to read them. There is at least 1/8 of books that I would love to read. But I don't have the time to actually pull it off, even if I am not doing any social networking etc., but the amount is really huge. Maybe, for a startup, someone could index all the content, not only share it.

pilimi_anna 4y ago

For those following along, we just added the full set of torrents to our TOR website.