21 ms·

Archivists Are Trying to Make Sure LibGen Never Goes Down

908 points by legatus 7y ago

dooglius 7y ago
There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.
- Mediterraneo10 7y ago
  This is a downside of Libgen: duplicate uploads, missing or erroneous metadata. You start wishing that there was at least some curation of the collection, so it could approach the quality of an academic library catalogue as many users are usedto. But I guess the people behind Libgen want to keep the number of people with database edit rights small. (When you upload a book, you yourself can edit the metadata for that book for 24 hours, but you cannot go through the rest of LibGen's database and make corrections.)
  - jplayer01 7y ago
    Maybe they should consider a system where users can suggest tags/metadata or flag erroneous data that can be reviewed and allowed by a select few?
    - Avamander 7y ago
      Integration with BookBrainz would be nice. The Brainz projects already consist of massive amounts of metadata curation and it would be possible to transfer that knowledge a bit.
- legatus 7y ago
  There are groups behind data curation as well, though it is much harder. LibGen sees an addition rate of about 230 GBs per month, while SciMag's is around 1.10 TBs per month. We should expect those numbers to increase in the future. The man-hours required to curate those database may very well cost much more than the storage and bandwidth required to store duplicates and incorrectly tagged files. In any case, as I said, there are people seriously interested in curating the LibGen database, though most efforts I know of are still in the earliest stages.
  - agumonkey 7y ago
    Do you know if they process PDF to reduce file size ?
    - guidoism 7y ago
      A lot of the data is in the djvu format which is very efficient for scanned books.
- Invictus0 7y ago
  I think the duplication issue is probably overstated. I doubt tackling that would shave off more than 20% of the total backup size.
  - dooglius 7y ago
    Speaking from personal experience, I usually see several results for any search. Granted, there's a big selection bias there, but 20% seems way too small.
    - sgillen 7y ago
      To be fair for textbooks at least I often see several results but often of different editions (1x edition 1, 2x edition 2, 1x edition 3 etc.). In some cases I think it's worthwhile keeping the different additions around, unless it becomes a huge burden.
    - abdullahkhalids 7y ago
      Because you or anyone is most likely to search for relatively popular books. So those books will have a multiple copies. But for every popular book, there are many unpopular, but still useful books, that only have a single copy.
    - MiroF 7y ago
      Usually the different results have meaningful differences - often times different edition or translator etc
    - asdff 7y ago
      In my experience it's different editions or mirrors.
  - throwaway894345 7y ago
    It's probably more of a nuisance for people wanting to use the content. E.g., copies with different metadata or tags.
  - driverdan 7y ago
legatus 7y ago
This is an extremely important effort. The LibGen archive contains around 32 TBs of books (by far the most common being scientific books and textbooks, with a healthy dose of non-STEM). The SciMag archive, backing up Sci-Hub, clocks in at around 67 TBs [0]. This is invaluable data that should not be lost. If you want to contribute, here's a few ways to do so. If you wish to donate bandwidth or storage, I personally know of at least a few mirroring efforts. Please get in touch with me over at legatusR(at)protonmail(dot)com and I can help direct you towards those behind this effort. If you don't have storage or bandwidth available, you can still help. Bookwarrior has requested help [1] in developing an HTTP-based decentralizing mechanism for LibGen's various forks. Those with experience in software may help make sure those invaluable archives are never lost. Another way of contributing is by donating bitcoin, as both LibGen [2] and The-Eye [3] accept donations. Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database. In any case, this effort has a noble goal, and I believe people of this community can contribute. P.S. The "Pirate Bay of Science" is actually LibGen, and I favor a title change (I posted it this way as to comply with HN guidelines). [0] http://185.39.10.101/stat.php http://185.39.10.101/stat.php [1] https://imgur.com/a/gmLB5pm https://imgur.com/a/gmLB5pm [2] bitcoin:12hQANsSHXxyPPgkhoBMSyHpXmzgVbdDGd?label=libgen, as found at http://185.39.10.101/ http://185.39.10.101/, listed in https://it.wikipedia.org/wiki/Library_Genesis https://it.wikipedia.org/wiki/Library_Genesis [3] Bitcoin address 3Mem5B2o3Qd2zAWEthJxUH28f7itbRttxM, as found in https://the-eye.eu/donate/ https://the-eye.eu/donate/. You can also buy merchandising from them at https://56k.pizza/ https://56k.pizza/.
- canuckintime 7y ago
  > Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database. There's no easy solution for scanning physical books, is there?
  - toomuchtodo 7y ago
    There are providers [1] that will destructively scan the book for you and return a PDF. If you want to preserve the book, you're stuck using a scanning rig [2]. The Internet Archive will also non-destructively scan as part of Open Library [3], but they only permit one checkout at a time of scanned works, and the latency can be high between sending them a book and it becoming available. FYI, 600 DPI is preferred for archival purposes. [1] http://1dollarscan.com/ http://1dollarscan.com/ (no affiliation, just a satisfied customer, can't scan certain textbooks due to publisher threats of litigation) [2] https://www.diybookscanner.org/ https://www.diybookscanner.org/ [3] https://openlibrary.org/help/faq https://openlibrary.org/help/faq
buboard 7y ago
one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage
- saalweachter 7y ago
  Do we have anything rated for a few millennia of interstellar radiation besides etched gold plates?
  - bangonkeyboard 7y ago
    Glass: https://news.microsoft.com/innovation-stories/ignite-project-silica-superman/ https://news.microsoft.com/innovation-stories/ignite-project..., https://en.wikipedia.org/wiki/5D_optical_data_storage https://en.wikipedia.org/wiki/5D_optical_data_storage
  - legatus 7y ago
    Microsoft's project Silica [0] may hopefully provide really long term, large capacity archive grade storage on earth. I wonder what effects interstellar radiation has on them. [0] https://www.theverge.com/2019/11/4/20942040/microsoft-project-silica-glass-storage-warner-bros-features-details https://www.theverge.com/2019/11/4/20942040/microsoft-projec...
    - kortex 7y ago
      Glass is pretty inert, full stop. It would depend on the voxel size but I imagine as long as you have more than a few hundered atoms per voxel/bit you will have survivability on the order of millenia, even in high radiation environments. Someone would have to do the nuclear cross section calculations to get a real bit error rate but glass is very tough stuff.
  - buboard 7y ago
    whats wrong with etched gold disks?
    - 7y ago
voldacar 7y ago
Is there a way to just download the whole 32TB to your own machine? I see a ton of mirrors but the content seems to be highly fragmented between them
- legatus 7y ago
  There are ways to do so. The archive is made up of many, many torrents (I believe it's a monthly if not biweekly update of the database). If you have the storage/bandwidth availability for the whole 32TBs, please get in touch and I may be able to help you get the whole deal without too much hassle. Otherwise, just pick some torrents (it would be best to pick them based on torrent health, but they are so many to check manually) and try to keep seeding as much as possible. EDIT: To find libgen's torrents health, check out this google sheet: https://docs.google.com/spreadsheets/d/1hqT7dVe8u09eatT93V2xvth-fUfNDxjE9SGT-KjLCj0/edit?usp=sharing https://docs.google.com/spreadsheets/d/1hqT7dVe8u09eatT93V2x... Thanks frgtpsswrdlame for the heads up.
  - voldacar 7y ago
    Thanks! I don't have 32TB free locally at the moment but I might soon. If and when that happens, I'll get in touch :)
  - frgtpsswrdlame 7y ago
    Actually there is now a google sheet which shows the health of the torrents so it should be easy to pick the most helpful torrents. It's linked in this post: reddit.com/e3yl23
  - ihuman 7y ago
    I'm pretty surprised by the lack of seeders. Out of the 2438 torrents listed, a third have 0 seeders, another third have 1 seeder, and all but 5 have less than 10. Hopefully the publicity boosts those numbers.
    - penagwin 7y ago
      From what I've heard a good chunk of people rotate their seeds for LibGen because their seedboxes can't handle all the connections for every torrent at once.
mister_hn 7y ago
One could use FAANG data centers to host them for free, it would be really great
- woofcat 7y ago
  Look at the google books project. That got shutdown real hard due to copyright issues and litigation after they invested a ton of money in digitizing some of the most valuable library collections in the world.
  - lovecg 7y ago
    It’s incredibly sad: https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/ https://www.theatlantic.com/technology/archive/2017/04/the-t...
    - guidoism 7y ago
      There are also multiple petabytes of microfiche scans of old newspapers. And of course nobody cares about it. The problem was shut down 2011ish and the data became "owned" by a team that didn't care for it. There was talk of just deleting the data because the team didn't want to pay for it. Ugh.
      - est31 7y ago
        Huh interesting. Have you got any details? Names of the project? There are probably institutions willing to host that content.
    - jandrese 7y ago
      I'm incredibly thankful that the public library system was invented before copyright maximalists got control of Congress.
      - BlueTemplar 7y ago
        In this case the issue seems to have come from "copyright minimalists" instead : wanting the books to be freely available, rather than making money for Google... I wonder why the Copyright Office didn't just buy Google Books, would only have cost a few hundred million $ ?
Avamander 7y ago
Why not publish the site over IPFS, that would make P2P hosting much simpler?
- legatus 7y ago
  Currently (at least for the-eye) it's about IPFS's barrier of entry. I expect LibGen's case to be similar. Most people don't know about it, and if even those that knew about it had to learn how IPFS works etc, they would probably just try to find the book they're looking for elsewhere.
  - Avamander 7y ago
    True, I too find it not ideal, but having such a massive library available over it surely would increase the interest in lowering the barrier of entry?
  - archi42 7y ago
    I am not fully aware how IPFS operates, but wouldn't it at least solve the back-end mirroring? Front-end servers would then "only" need to access IPFS for continuous syncing of metadata (for search) and fetching user-requested files (upon request).
    - deleted 7y ago
      [deleted]
  - wgjordan 7y ago
    No need to conflate the frontend (the end-user interface that 'most people' use when trying to 'find the book they're looking for') with the mirroring/archiving backend (the distributed/p2p technology used to 'make sure LibGen never goes down'). The frontend would still be a user-friendly HTTP web-application (or collection of several) that pulls (portions of) the archive from the distributed/resilient backend to serve individual files to clients. The backend can be a relatively obscure, geeky, post-BitTorrent p2p software like IPFS or Dat, as long as those willing to donate bandwidth/storage can run it on their systems. This is a vastly different audience from 'most people'. The real question is which software's features best fits the backend use-case (efficiently hosting a very large and growing/evolving, IP-infringing dataset). Dat [1] has features to (1) update data and efficiently synchronize changes, and to (2) efficiently provide random-access data from larger datasets. Two quite compelling advancements over BitTorrent for this use-case. [1] https://docs.datproject.org/docs/faq#how-is-dat-different-than-academic-torrents-or-bittorrent https://docs.datproject.org/docs/faq#how-is-dat-different-th...
Tepix 7y ago
Related: Looking at harddisk cost per terabyte, quite often extern drives are cheaper than internal ones. For example right now in Germany I can get a WD 8TB USB 3.0 drive for 135€ but the cheapest internal 8TB drive costs 169€. Any idea why? It's puzzling.
- cr0sh 7y ago
  My best guess would be that more people buy external drives than internal, and those that "manufacture" external drives (ie - buy internal drives and repackage them) purchase in larger volumes than those that sell bare internal drives. Of course, that wouldn't explain the difference between a WD external drive and that same drive as an internal drive - assuming that WD actually manufactures both (and doesn't just license the name provided the 3rd party uses their drives)...
- LameRubberDucky 7y ago
  I noticed this yesterday while shopping for cyber Monday deals. If you want to load up a server with drives, perhaps the external drives can be removed from their cases and used internally?
  - javitury 7y ago
    Check it before you buy it. Years ago I bought a 1TB WD external drive. The usb interface was connected directly to the drive.
    - LameRubberDucky 7y ago
      Thanks! I will have to read up on shucking and and /r/DataHoarder. I would think someone already has a list of which external drives can be used this way.
  - xur17 7y ago
    I did exactly this when setting up a NAS. Saved $50 / drive, and it took me a few minutes to remove the drive from each.
  - 7y ago
miki123211 7y ago
The new architecture of pirate sites, what I call the Hydra architecture, seems pretty interesting to me. There isn't a single site hosting the content, but a group of mirrors freely exchanging data between one another. In case some of them go down, the other ones still remain and new ones can appear, copying data from the remaining mirrors. This is like a hydra that grows two heads every time you chop one off. It's absolutely unkillable, as there's no single group or server to sue. A more advanced version of this architecture is used by pirate addons for the Kodi media center software. Basically, you have a bunch of completely legal and above board services like Imdb that contain video metadata. They provide the search results, the artworks, the plot descriptions, episode lists for TV shows etc. Impossible to sue and shut down, as they're legal. Then, you have a large number of illegal services that, essentially, map IDs from websites like IMDB to links. Those links lead to websites like Openload, which let you host videos. They're in the gray area, if they comply with DMCA requests and are in a reasonably safe jurisdiction, they're unlikely to be shut down. On the Kodi side, you have a bunch of addons. There are the legitimate ones that access IMDB and give you the IDs, the not that legitimate ones that map IDs to URLs, and the half-legitimate ones that can actually play stuff ron those URLS (not an easy taks, as websites usually try to prevent you from playing something without seeing their ads). Those addons are distributed as libraries, and are used as dependencies by user-friendly frontends. Those frontends usually depend on several addons in each category, so, in case one goes down, all the other ones still remain. It's all so decentralized and ownerless that there's no single point of failure. The best you can do is killing the frontend addon, but it's easy to make a new one, and users are used to switching them every few months.
- deleted 7y ago
  [deleted]
- bobongo 7y ago
  > there's no single point of failure. The best you can do is killing the frontend addon Single decentralized service, providing access to all content, national and international, free of DRM, for all platforms, for a proper, fair, and non-monopolist price. That will pull all the users who are willing to pay for content over to the paid service, and those who remained were not willing to pay regardless of what you did anyhow.
sanxiyn 7y ago
Yongle Encyclopedia was a similar project of the 15th century China. It was the largest encyclopedia in the world for 600 years until surpassed by Wikipedia. Alas, Yongle Encyclopedia is almost completely lost now. Archiving is harder than you think. https://en.wikipedia.org/wiki/Yongle_Encyclopedia https://en.wikipedia.org/wiki/Yongle_Encyclopedia
- weinzierl 7y ago
  I read the Wikipedia article about it and the sad thing is that the majority of the Yongle Encyclopedia seem to have been destroyed only in quite recent times.
  - knolax 7y ago
    > but 90 percent of the 1567 manuscript survived until the Second Opium War in the Qing dynasty. In 1860, the Anglo-French invasion of Beijing resulted in extensive burning and looting of the city,[16] with the British and French soldiers taking large portions of the manuscript as souvenirs. Preservation is easy if you don't get invaded.
    - asdff 7y ago
      It's easy if you anticipate these things. Who put the dead sea scrolls in that cave in the middle of nowhere? Not someone who went in and forgot their scroll one day. Someone who had the foresight that this would be a safe space in the face of who knows what future threat. And it payed off.
      - zaarn 7y ago
        I doubt it was that intentional. I would wager that "someone forgot about it" is the more likely explanation.
- 8bitsrule 7y ago
  WP says that it was never printed for the general public. Hmmm. Had it been (parts duplicated, say, at hundreds of sites), most of it would probably have survived.
asdernr 7y ago
If only some of the money made would reach the scientists lel. Most of em will give you their paper per mail if you aak them. The majority does not want them to sit behind paywalls...
FpUser 7y ago
I did not know about LibGen until this post. Too bad for me living in a cave. Anyways this is amazing project. Best luck to them and similar efforts.
burtonator 7y ago
What's interesting is that 32TB is becoming more and more affordable and the research material is roughly staying about the same size. That might change though as people start including video + data within papers and have new notebook formats that are live and contain docker containers/ipython, etc. It's a shame we can't just mail these around.
- deleted 7y ago
  [deleted]
- jbverschoor 7y ago
  You can buy 48TB (4x12TB) for €1000. Store some index on an SSD, and you have another full node.
  - washadjeffmad 7y ago
    If you don't care about warranty, 8 and 12TB drives routinely go for $15/TB on sale inside WD Elements. I picked up 32TB for just under $500 with discount over the holiday that way.
    - ancientworldnow 7y ago
      Even if you shuck the drive, as long as you keep the enclosure Western Digital will still honor the warranty.
      - iwalton3 7y ago
        I’ve heard that you can send the drives back without the enclosure and they have still honored the warranty. https://www.reddit.com/r/DataHoarder/comments/am9vdv/easystore_warranty_experiment_success/ https://www.reddit.com/r/DataHoarder/comments/am9vdv/easysto...
    - StavrosK 7y ago
      Can you elaborate? What's the catch?
fghtr 7y ago
Are there any i2p torrents? I guess anonymity might be helpful if I want to mirror/seed this data...
- zozbot234 7y ago
  I assume anyone could simply seed the "official" torrents via i2p? Not sure how that system actually works, it's interesting for sure but a lot less well-known than the alternatives.
burtonator 7y ago
I've thought that we could potentially build an end to end encrypted datastore within Polar and possibly add IPFS support to potentially help with this issue. Here's a blog post about our datastores for some background. https://getpolarized.io/2019/03/22/portable-datastores-and-platform-independence.html https://getpolarized.io/2019/03/22/portable-datastores-and-p... ... essentially Polar is a PDF manager and knowledge repository for academics, scientists, intellectuals, etc. One secondary challenge we have is allowing for sharing of research but I'd like to do it in a secure and distributed manner. Some of our users are concerned about their eBooks being stored unencrypted and while for the majority of our users this will never be a problem I can see this being an issue in countries with political regimes that are hostile to open research. In the US we have an issue of researchers being harassed over climate change btw. Having a way to encrypt your knowledge repository (ebooks) would help academic freedom as your employer or government couldn't force you to give them your repository. But what if we went beyond this and provided a way to ADD documents to the repository from a site like LibGen? Then we'd have the ability to easily, with one click, encrypt the document (end to end) and added it to our repository. If we can add support for Polar to allow colleagues to share directly, this would be a virtual mirror of LibGen. Alice could add books b1, b2, b3 to their repo, they could then share with Bob, only he would be able to see b1, b2, b3, then they would generate a shared symmetric key to share the books. No 3rd party (including me) would have any knowledge what's going on. I'm going to assume our users are not going to do anything nefarious or pirate any books. I'm also certain that they're confirming to the necessary laws ... The challenge though is that while we'd be able to have a mirror of LibGen and more material, it would be a probabilistic mirror - I'm sure we'd have like 60% of it but the obscure material wouldn't be mirrored. Right now our datastores support just local disk, and Firebase (which is Google Cloud basically). While we would encrypt the data end to end in Google Cloud I can totally understand why users might not like to use that platform. One major issue is China where it's blocked. Something like IPFS could go a long way to solving this but it's still very new and I haven't hacked on it much.
turc1656 7y ago
I don't see anyone having mentioned the possibility of posting this data to Usenet at all - at minimum for archival purposes which should be good for ~8-9 years. That way at least the data isn't lost. With so many of those torrents have 0 or 1 seed, this is a serious risk I think, despite the comments elsewhere about people rotating what they seed. I realize that doesn't solve the access problem for most people as most of the users who need this research might not know how to use usenet or even be familiar with it at all, but I think the first major concern would be to secure the entire repository on a stable network. Usenet seems like a good place for that even if it doesn't serves as a means of distribution. Encrypting the uploads would make them immune to DMCA takedowns provided that the decryption keys weren't made public and were only shared with individuals related to the maintenance of the LibGen project.
- walrus01 7y ago
  Two thoughts on that. Encoding it to a text format with CRC data for posting to usenet is highly inefficient in terms of data storage. And 33TB of stuff is not going to be retained for 8-9 years, the last I checked due to the huge volume of binaries traffic, the major commercial usenet feed providers have at most 6-9 months of retention for the major binary groups. Beyond that it becomes cost prohibitive for them in terms of disk storage requirements. This is not an issue for the majority of their customers, 6-9 months is more than long enough retention to go find a 40GB 2160p copy of some recently-released-on-bluray movie.
  - turc1656 7y ago
    Entirely agree about the lack of efficiency. No question about that. However, in my personal experience, I have seen no issues downloading old data from any binary group. At least not with the provider I have. In fact, just this past week I obtained something sizable (several GBs) with no damaged parts so didn't even need the parchive recovery files at all. This has always been my experience. I've never seen anything like the pruning you are talking about. That sounds more like an issue with your specific provider to me.
  - trevyn 7y ago
EthanHeilman 7y ago
Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material became available. All joking aside, I do wonder wither digital or analogue formats are better able to survive into the distant future. * What impact will DRM have on the accessibility of our knowledge to future historians? * Is anything recoverable from a harddrive or flash media after 500 years in a landfill? * Will compressed files be more of less recoverable? What about git archives? * Will the future know the shape of our plastic GI Joes toys but not the content of the GI Joes cartoon?
- qmmmur 7y ago
  Pretty much everyone in a tech job could afford to buy 40TB of storage at home, or remotely and mirror the entire repo. I think that given this low barrier of entry if you can afford to help preserve the information then you can and probably should. Even if a small amount do it it's more points of recovery.
  - int_19h 7y ago
    Storage isn't hard, but downloading 40 Tb can be a problem. Are there any arrangements for physical distribution (of the "truck loaded with USB drives" variety)?
  - asdefghyk 7y ago
    Id say.... in the day .... anyone could afford to buy a single floppy disk and store files on it. But how many actually did and how many are actually recoverable. Lots probably got thrown out in intervening years.
- unicornporn 7y ago
  In the GLAM sector the LOCKSS[1] is project is quite well-known. It tries to deal with some of the resiliency problems that is inherent in digital preservation. However, I'd guess this system does not offer the needed anonymity. [1] https://www.lockss.org/ https://www.lockss.org/ ; https://en.wikipedia.org/wiki/LOCKSS https://en.wikipedia.org/wiki/LOCKSS
nullifidian 7y ago
Posting that here only creates problems for them. The more it's known in the west the more likely it will go down.
- coffee12345 7y ago
  +1, bookwarrior has warned about this.
  - news_hacker 7y ago
    who is bookwarrior?
skjoldr 7y ago
Can't Tahoe-LAFS help with this kind of a challenge? I don't have experience with it, but it looks stable.
knzhou 7y ago
Libgen is one of the greatest contributors to scientific productivity worldwide, possibly beaten only by Sci-Hub. Just about everybody in academia knows about it. If it ever vanished, some of us could probably still get by trading files from person to person, but nothing could be as perfect as what we got now.
- lioeters 7y ago
  > possibly beaten only by Sci-Hub Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source. So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?). > Just about everybody in academia knows about it. That really says something about the state of society, this tension between copyright laws (and the motivations behind them) and the intellectual ideal of free and open access to knowledge.
  - mikorym 7y ago
    I am not an expert on the topic, but I believe that in the former Soviet Union it was common between mathematicians to pass around preprints (a la arXiv). These then perculated through to the West. I think it had to do with the USSR and their restrictive (if we are being euphemistic) policies towards academics.
    - gdy 7y ago
      "the USSR and their restrictive (if we are being euphemistic) policies towards academics." What do you mean?
      - alteria 7y ago
        Their policies were for more than "restrictive" is how I'm reading it See [1] [1] https://en.wikipedia.org/wiki/Suppressed_research_in_the_Soviet_Union https://en.wikipedia.org/wiki/Suppressed_research_in_the_Sov...
boksiora 7y ago
its best to split on small torrents on few 1-2 GB so normal users can seed
milofeynman 7y ago
I'd like to dedicate 1TB of my FreeNAS to something like this. Would be nice to run a small container with some P2P service that contained that chunk.
whydoyoucare 7y ago
Isn't scanning a physical book and uploading a soft-copy, a landmine of hazards (both legal and moral)? Essentially you are encouraging (some) unlawful activity... I am not so sure I am onboard with this idea!
- guidoism 7y ago
  It’s easy to take this stance in a rich country. But what about the people in countries where one of these books cost the equivalent of a year’s wages. Not so black and white eh?
  - whydoyoucare 7y ago
    As far as I know, prices of books differ between rich and developing nations. For e.g., The C Programming Language that costs $50 in the US [1], is sold for Rs. 259 (~$4 US) in India. I believe that is the case with most "economy editions" specifically targeted at developing nations. It certainly isn't an "year's wages". While I do understand your point, it still does not justify encouraging modern-day Robinhoods' and breaking the law. [1] https://www.amazon.com/Programming-Language-2nd-Brian-Kernighan/dp/0131103628 https://www.amazon.com/Programming-Language-2nd-Brian-Kernig... [2] https://www.amazon.in/Programming-Language-Kernighan-Dennis-Ritchie/dp/9332549443/ https://www.amazon.in/Programming-Language-Kernighan-Dennis-...
    - guidoism 7y ago
      Economy editions are nice, but only make up a tiny fraction of what's out there to consume. Again, $4 is great for a relatively rich nation like India, but what about Eritrea? Maybe that was a little flippant, but The Law in a big rich country is pretty meaningless to someone trying to make a better life for themselves in a poor country.
    - esotericn 7y ago
      Hi. You're speaking to someone from the UK. When I grew up we had to walk uphill both ways, to quote the Four Yorkshiremen sketch. I couldn't afford to buy things like books, operating systems, games, etc. We spent the money on food, rent, and on the Internet. People like that past version of me are going to pirate regardless of what you say. Law breaking? You can't see me grinning ear to ear. I like to think my tax bill makes up for that.
sandov 7y ago
Let me say this: I fucking love libgen. It actually makes my life better and I'm so thankful to the people running it.
mutant 7y ago
I'd say IPFS, but That's a pretty big commitment from an entire community to keep alive.
lukebuehler 7y ago
To me, an aspiring scholar, LibGen is the most amazing tool ever. Things like inter-library loan and access to databases on university networks already make life so much easier to what it used to be—but nothing beats LibGen in terms of convenience. I’m in a the nowadays obscure field of patristic theology and I can’t believe how much stuff I can find on LibGen, often things that even highly specialized research libraries like Harvard don’t have. The hours that LibGen saved me in gathering all the sources for my research must be in the hundreds. Thank you!
6510 7y ago
Imagine this: - A tiny well behaved client that starts with the OS. - It downloads rare bits of the archive at 1 kb/s obtaining 1 GB every 278 hours. It should stop around 100 MB to 5 GB. - It periodically announces what chunks/documents it has. - It seeds those chunks at 1 kb/s - Chunks/documents that have thousands of seeds already are not announced. Eventually those are pruned. This escalates the situation to the point where everyone can help without it costing anything. If someone is trying to obtain a 20 mb pfd it would take 5 and a half hours using a single 1 kb seed. With just 50 seeds it's just 8 min.

908 points by legatus 7y ago

There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.

Mediterraneo10 7y ago
This is a downside of Libgen: duplicate uploads, missing or erroneous metadata. You start wishing that there was at least some curation of the collection, so it could approach the quality of an academic library catalogue as many users are usedto. But I guess the people behind Libgen want to keep the number of people with database edit rights small. (When you upload a book, you yourself can edit the metadata for that book for 24 hours, but you cannot go through the rest of LibGen's database and make corrections.)
- jplayer01 7y ago
  Maybe they should consider a system where users can suggest tags/metadata or flag erroneous data that can be reviewed and allowed by a select few?
  - Avamander 7y ago
    Integration with BookBrainz would be nice. The Brainz projects already consist of massive amounts of metadata curation and it would be possible to transfer that knowledge a bit.
legatus 7y ago
There are groups behind data curation as well, though it is much harder. LibGen sees an addition rate of about 230 GBs per month, while SciMag's is around 1.10 TBs per month. We should expect those numbers to increase in the future. The man-hours required to curate those database may very well cost much more than the storage and bandwidth required to store duplicates and incorrectly tagged files. In any case, as I said, there are people seriously interested in curating the LibGen database, though most efforts I know of are still in the earliest stages.
- agumonkey 7y ago
  Do you know if they process PDF to reduce file size ?
  - guidoism 7y ago
    A lot of the data is in the djvu format which is very efficient for scanned books.
Invictus0 7y ago
I think the duplication issue is probably overstated. I doubt tackling that would shave off more than 20% of the total backup size.
- dooglius 7y ago
  Speaking from personal experience, I usually see several results for any search. Granted, there's a big selection bias there, but 20% seems way too small.
  - sgillen 7y ago
    To be fair for textbooks at least I often see several results but often of different editions (1x edition 1, 2x edition 2, 1x edition 3 etc.). In some cases I think it's worthwhile keeping the different additions around, unless it becomes a huge burden.
  - abdullahkhalids 7y ago
    Because you or anyone is most likely to search for relatively popular books. So those books will have a multiple copies. But for every popular book, there are many unpopular, but still useful books, that only have a single copy.
  - MiroF 7y ago
    Usually the different results have meaningful differences - often times different edition or translator etc
  - asdff 7y ago
    In my experience it's different editions or mirrors.
- throwaway894345 7y ago
  It's probably more of a nuisance for people wanting to use the content. E.g., copies with different metadata or tags.
- driverdan 7y ago

legatus 7y ago

This is an extremely important effort. The LibGen archive contains around 32 TBs of books (by far the most common being scientific books and textbooks, with a healthy dose of non-STEM). The SciMag archive, backing up Sci-Hub, clocks in at around 67 TBs [0]. This is invaluable data that should not be lost. If you want to contribute, here's a few ways to do so. If you wish to donate bandwidth or storage, I personally know of at least a few mirroring efforts. Please get in touch with me over at legatusR(at)protonmail(dot)com and I can help direct you towards those behind this effort. If you don't have storage or bandwidth available, you can still help. Bookwarrior has requested help [1] in developing an HTTP-based decentralizing mechanism for LibGen's various forks. Those with experience in software may help make sure those invaluable archives are never lost. Another way of contributing is by donating bitcoin, as both LibGen [2] and The-Eye [3] accept donations. Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database. In any case, this effort has a noble goal, and I believe people of this community can contribute. P.S. The "Pirate Bay of Science" is actually LibGen, and I favor a title change (I posted it this way as to comply with HN guidelines). [0] http://185.39.10.101/stat.php http://185.39.10.101/stat.php [1] https://imgur.com/a/gmLB5pm https://imgur.com/a/gmLB5pm [2] bitcoin:12hQANsSHXxyPPgkhoBMSyHpXmzgVbdDGd?label=libgen, as found at http://185.39.10.101/ http://185.39.10.101/, listed in https://it.wikipedia.org/wiki/Library_Genesis https://it.wikipedia.org/wiki/Library_Genesis [3] Bitcoin address 3Mem5B2o3Qd2zAWEthJxUH28f7itbRttxM, as found in https://the-eye.eu/donate/ https://the-eye.eu/donate/. You can also buy merchandising from them at https://56k.pizza/ https://56k.pizza/.

canuckintime 7y ago
> Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database. There's no easy solution for scanning physical books, is there?
- toomuchtodo 7y ago
  There are providers [1] that will destructively scan the book for you and return a PDF. If you want to preserve the book, you're stuck using a scanning rig [2]. The Internet Archive will also non-destructively scan as part of Open Library [3], but they only permit one checkout at a time of scanned works, and the latency can be high between sending them a book and it becoming available. FYI, 600 DPI is preferred for archival purposes. [1] http://1dollarscan.com/ http://1dollarscan.com/ (no affiliation, just a satisfied customer, can't scan certain textbooks due to publisher threats of litigation) [2] https://www.diybookscanner.org/ https://www.diybookscanner.org/ [3] https://openlibrary.org/help/faq https://openlibrary.org/help/faq

buboard 7y ago

one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage

saalweachter 7y ago
Do we have anything rated for a few millennia of interstellar radiation besides etched gold plates?
- bangonkeyboard 7y ago
  Glass: https://news.microsoft.com/innovation-stories/ignite-project-silica-superman/ https://news.microsoft.com/innovation-stories/ignite-project..., https://en.wikipedia.org/wiki/5D_optical_data_storage https://en.wikipedia.org/wiki/5D_optical_data_storage
- legatus 7y ago
  Microsoft's project Silica [0] may hopefully provide really long term, large capacity archive grade storage on earth. I wonder what effects interstellar radiation has on them. [0] https://www.theverge.com/2019/11/4/20942040/microsoft-project-silica-glass-storage-warner-bros-features-details https://www.theverge.com/2019/11/4/20942040/microsoft-projec...
  - kortex 7y ago
    Glass is pretty inert, full stop. It would depend on the voxel size but I imagine as long as you have more than a few hundered atoms per voxel/bit you will have survivability on the order of millenia, even in high radiation environments. Someone would have to do the nuclear cross section calculations to get a real bit error rate but glass is very tough stuff.
- buboard 7y ago
  whats wrong with etched gold disks?
  - 7y ago

voldacar 7y ago

Is there a way to just download the whole 32TB to your own machine? I see a ton of mirrors but the content seems to be highly fragmented between them

legatus 7y ago
There are ways to do so. The archive is made up of many, many torrents (I believe it's a monthly if not biweekly update of the database). If you have the storage/bandwidth availability for the whole 32TBs, please get in touch and I may be able to help you get the whole deal without too much hassle. Otherwise, just pick some torrents (it would be best to pick them based on torrent health, but they are so many to check manually) and try to keep seeding as much as possible. EDIT: To find libgen's torrents health, check out this google sheet: https://docs.google.com/spreadsheets/d/1hqT7dVe8u09eatT93V2xvth-fUfNDxjE9SGT-KjLCj0/edit?usp=sharing https://docs.google.com/spreadsheets/d/1hqT7dVe8u09eatT93V2x... Thanks frgtpsswrdlame for the heads up.
- voldacar 7y ago
  Thanks! I don't have 32TB free locally at the moment but I might soon. If and when that happens, I'll get in touch :)
- frgtpsswrdlame 7y ago
  Actually there is now a google sheet which shows the health of the torrents so it should be easy to pick the most helpful torrents. It's linked in this post: reddit.com/e3yl23
- ihuman 7y ago
  I'm pretty surprised by the lack of seeders. Out of the 2438 torrents listed, a third have 0 seeders, another third have 1 seeder, and all but 5 have less than 10. Hopefully the publicity boosts those numbers.
  - penagwin 7y ago
    From what I've heard a good chunk of people rotate their seeds for LibGen because their seedboxes can't handle all the connections for every torrent at once.

mister_hn 7y ago

One could use FAANG data centers to host them for free, it would be really great

woofcat 7y ago
Look at the google books project. That got shutdown real hard due to copyright issues and litigation after they invested a ton of money in digitizing some of the most valuable library collections in the world.
- lovecg 7y ago
  It’s incredibly sad: https://www.theatlantic.com/technology/archive/2017/04/the-tragedy-of-google-books/523320/ https://www.theatlantic.com/technology/archive/2017/04/the-t...
  - guidoism 7y ago
    There are also multiple petabytes of microfiche scans of old newspapers. And of course nobody cares about it. The problem was shut down 2011ish and the data became "owned" by a team that didn't care for it. There was talk of just deleting the data because the team didn't want to pay for it. Ugh.
    - est31 7y ago
      Huh interesting. Have you got any details? Names of the project? There are probably institutions willing to host that content.
  - jandrese 7y ago
    I'm incredibly thankful that the public library system was invented before copyright maximalists got control of Congress.
    - BlueTemplar 7y ago
      In this case the issue seems to have come from "copyright minimalists" instead : wanting the books to be freely available, rather than making money for Google... I wonder why the Copyright Office didn't just buy Google Books, would only have cost a few hundred million $ ?

Avamander 7y ago

Why not publish the site over IPFS, that would make P2P hosting much simpler?

legatus 7y ago
Currently (at least for the-eye) it's about IPFS's barrier of entry. I expect LibGen's case to be similar. Most people don't know about it, and if even those that knew about it had to learn how IPFS works etc, they would probably just try to find the book they're looking for elsewhere.
- Avamander 7y ago
  True, I too find it not ideal, but having such a massive library available over it surely would increase the interest in lowering the barrier of entry?
- archi42 7y ago
  I am not fully aware how IPFS operates, but wouldn't it at least solve the back-end mirroring? Front-end servers would then "only" need to access IPFS for continuous syncing of metadata (for search) and fetching user-requested files (upon request).
  - deleted 7y ago
    [deleted]
- wgjordan 7y ago
  No need to conflate the frontend (the end-user interface that 'most people' use when trying to 'find the book they're looking for') with the mirroring/archiving backend (the distributed/p2p technology used to 'make sure LibGen never goes down'). The frontend would still be a user-friendly HTTP web-application (or collection of several) that pulls (portions of) the archive from the distributed/resilient backend to serve individual files to clients. The backend can be a relatively obscure, geeky, post-BitTorrent p2p software like IPFS or Dat, as long as those willing to donate bandwidth/storage can run it on their systems. This is a vastly different audience from 'most people'. The real question is which software's features best fits the backend use-case (efficiently hosting a very large and growing/evolving, IP-infringing dataset). Dat [1] has features to (1) update data and efficiently synchronize changes, and to (2) efficiently provide random-access data from larger datasets. Two quite compelling advancements over BitTorrent for this use-case. [1] https://docs.datproject.org/docs/faq#how-is-dat-different-than-academic-torrents-or-bittorrent https://docs.datproject.org/docs/faq#how-is-dat-different-th...

Tepix 7y ago

Related: Looking at harddisk cost per terabyte, quite often extern drives are cheaper than internal ones. For example right now in Germany I can get a WD 8TB USB 3.0 drive for 135€ but the cheapest internal 8TB drive costs 169€. Any idea why? It's puzzling.

cr0sh 7y ago
My best guess would be that more people buy external drives than internal, and those that "manufacture" external drives (ie - buy internal drives and repackage them) purchase in larger volumes than those that sell bare internal drives. Of course, that wouldn't explain the difference between a WD external drive and that same drive as an internal drive - assuming that WD actually manufactures both (and doesn't just license the name provided the 3rd party uses their drives)...
LameRubberDucky 7y ago
I noticed this yesterday while shopping for cyber Monday deals. If you want to load up a server with drives, perhaps the external drives can be removed from their cases and used internally?
- javitury 7y ago
  Check it before you buy it. Years ago I bought a 1TB WD external drive. The usb interface was connected directly to the drive.
  - LameRubberDucky 7y ago
    Thanks! I will have to read up on shucking and and /r/DataHoarder. I would think someone already has a list of which external drives can be used this way.
- xur17 7y ago
  I did exactly this when setting up a NAS. Saved $50 / drive, and it took me a few minutes to remove the drive from each.
- 7y ago

miki123211 7y ago

The new architecture of pirate sites, what I call the Hydra architecture, seems pretty interesting to me. There isn't a single site hosting the content, but a group of mirrors freely exchanging data between one another. In case some of them go down, the other ones still remain and new ones can appear, copying data from the remaining mirrors. This is like a hydra that grows two heads every time you chop one off. It's absolutely unkillable, as there's no single group or server to sue. A more advanced version of this architecture is used by pirate addons for the Kodi media center software. Basically, you have a bunch of completely legal and above board services like Imdb that contain video metadata. They provide the search results, the artworks, the plot descriptions, episode lists for TV shows etc. Impossible to sue and shut down, as they're legal. Then, you have a large number of illegal services that, essentially, map IDs from websites like IMDB to links. Those links lead to websites like Openload, which let you host videos. They're in the gray area, if they comply with DMCA requests and are in a reasonably safe jurisdiction, they're unlikely to be shut down. On the Kodi side, you have a bunch of addons. There are the legitimate ones that access IMDB and give you the IDs, the not that legitimate ones that map IDs to URLs, and the half-legitimate ones that can actually play stuff ron those URLS (not an easy taks, as websites usually try to prevent you from playing something without seeing their ads). Those addons are distributed as libraries, and are used as dependencies by user-friendly frontends. Those frontends usually depend on several addons in each category, so, in case one goes down, all the other ones still remain. It's all so decentralized and ownerless that there's no single point of failure. The best you can do is killing the frontend addon, but it's easy to make a new one, and users are used to switching them every few months.

deleted 7y ago
[deleted]
bobongo 7y ago
> there's no single point of failure. The best you can do is killing the frontend addon Single decentralized service, providing access to all content, national and international, free of DRM, for all platforms, for a proper, fair, and non-monopolist price. That will pull all the users who are willing to pay for content over to the paid service, and those who remained were not willing to pay regardless of what you did anyhow.

sanxiyn 7y ago

Yongle Encyclopedia was a similar project of the 15th century China. It was the largest encyclopedia in the world for 600 years until surpassed by Wikipedia. Alas, Yongle Encyclopedia is almost completely lost now. Archiving is harder than you think. https://en.wikipedia.org/wiki/Yongle_Encyclopedia https://en.wikipedia.org/wiki/Yongle_Encyclopedia

weinzierl 7y ago
I read the Wikipedia article about it and the sad thing is that the majority of the Yongle Encyclopedia seem to have been destroyed only in quite recent times.
- knolax 7y ago
  > but 90 percent of the 1567 manuscript survived until the Second Opium War in the Qing dynasty. In 1860, the Anglo-French invasion of Beijing resulted in extensive burning and looting of the city,[16] with the British and French soldiers taking large portions of the manuscript as souvenirs. Preservation is easy if you don't get invaded.
  - asdff 7y ago
    It's easy if you anticipate these things. Who put the dead sea scrolls in that cave in the middle of nowhere? Not someone who went in and forgot their scroll one day. Someone who had the foresight that this would be a safe space in the face of who knows what future threat. And it payed off.
    - zaarn 7y ago
      I doubt it was that intentional. I would wager that "someone forgot about it" is the more likely explanation.
8bitsrule 7y ago
WP says that it was never printed for the general public. Hmmm. Had it been (parts duplicated, say, at hundreds of sites), most of it would probably have survived.

asdernr 7y ago

If only some of the money made would reach the scientists lel. Most of em will give you their paper per mail if you aak them. The majority does not want them to sit behind paywalls...

FpUser 7y ago

I did not know about LibGen until this post. Too bad for me living in a cave. Anyways this is amazing project. Best luck to them and similar efforts.

burtonator 7y ago

What's interesting is that 32TB is becoming more and more affordable and the research material is roughly staying about the same size. That might change though as people start including video + data within papers and have new notebook formats that are live and contain docker containers/ipython, etc. It's a shame we can't just mail these around.

deleted 7y ago
[deleted]
jbverschoor 7y ago
You can buy 48TB (4x12TB) for €1000. Store some index on an SSD, and you have another full node.
- washadjeffmad 7y ago
  If you don't care about warranty, 8 and 12TB drives routinely go for $15/TB on sale inside WD Elements. I picked up 32TB for just under $500 with discount over the holiday that way.
  - ancientworldnow 7y ago
    Even if you shuck the drive, as long as you keep the enclosure Western Digital will still honor the warranty.
    - iwalton3 7y ago
      I’ve heard that you can send the drives back without the enclosure and they have still honored the warranty. https://www.reddit.com/r/DataHoarder/comments/am9vdv/easystore_warranty_experiment_success/ https://www.reddit.com/r/DataHoarder/comments/am9vdv/easysto...
  - StavrosK 7y ago
    Can you elaborate? What's the catch?

fghtr 7y ago

Are there any i2p torrents? I guess anonymity might be helpful if I want to mirror/seed this data...

zozbot234 7y ago
I assume anyone could simply seed the "official" torrents via i2p? Not sure how that system actually works, it's interesting for sure but a lot less well-known than the alternatives.

burtonator 7y ago

I've thought that we could potentially build an end to end encrypted datastore within Polar and possibly add IPFS support to potentially help with this issue. Here's a blog post about our datastores for some background. https://getpolarized.io/2019/03/22/portable-datastores-and-platform-independence.html https://getpolarized.io/2019/03/22/portable-datastores-and-p... ... essentially Polar is a PDF manager and knowledge repository for academics, scientists, intellectuals, etc. One secondary challenge we have is allowing for sharing of research but I'd like to do it in a secure and distributed manner. Some of our users are concerned about their eBooks being stored unencrypted and while for the majority of our users this will never be a problem I can see this being an issue in countries with political regimes that are hostile to open research. In the US we have an issue of researchers being harassed over climate change btw. Having a way to encrypt your knowledge repository (ebooks) would help academic freedom as your employer or government couldn't force you to give them your repository. But what if we went beyond this and provided a way to ADD documents to the repository from a site like LibGen? Then we'd have the ability to easily, with one click, encrypt the document (end to end) and added it to our repository. If we can add support for Polar to allow colleagues to share directly, this would be a virtual mirror of LibGen. Alice could add books b1, b2, b3 to their repo, they could then share with Bob, only he would be able to see b1, b2, b3, then they would generate a shared symmetric key to share the books. No 3rd party (including me) would have any knowledge what's going on. I'm going to assume our users are not going to do anything nefarious or pirate any books. I'm also certain that they're confirming to the necessary laws ... The challenge though is that while we'd be able to have a mirror of LibGen and more material, it would be a probabilistic mirror - I'm sure we'd have like 60% of it but the obscure material wouldn't be mirrored. Right now our datastores support just local disk, and Firebase (which is Google Cloud basically). While we would encrypt the data end to end in Google Cloud I can totally understand why users might not like to use that platform. One major issue is China where it's blocked. Something like IPFS could go a long way to solving this but it's still very new and I haven't hacked on it much.

turc1656 7y ago

I don't see anyone having mentioned the possibility of posting this data to Usenet at all - at minimum for archival purposes which should be good for ~8-9 years. That way at least the data isn't lost. With so many of those torrents have 0 or 1 seed, this is a serious risk I think, despite the comments elsewhere about people rotating what they seed. I realize that doesn't solve the access problem for most people as most of the users who need this research might not know how to use usenet or even be familiar with it at all, but I think the first major concern would be to secure the entire repository on a stable network. Usenet seems like a good place for that even if it doesn't serves as a means of distribution. Encrypting the uploads would make them immune to DMCA takedowns provided that the decryption keys weren't made public and were only shared with individuals related to the maintenance of the LibGen project.

walrus01 7y ago
Two thoughts on that. Encoding it to a text format with CRC data for posting to usenet is highly inefficient in terms of data storage. And 33TB of stuff is not going to be retained for 8-9 years, the last I checked due to the huge volume of binaries traffic, the major commercial usenet feed providers have at most 6-9 months of retention for the major binary groups. Beyond that it becomes cost prohibitive for them in terms of disk storage requirements. This is not an issue for the majority of their customers, 6-9 months is more than long enough retention to go find a 40GB 2160p copy of some recently-released-on-bluray movie.
- turc1656 7y ago
  Entirely agree about the lack of efficiency. No question about that. However, in my personal experience, I have seen no issues downloading old data from any binary group. At least not with the provider I have. In fact, just this past week I obtained something sizable (several GBs) with no damaged parts so didn't even need the parchive recovery files at all. This has always been my experience. I've never seen anything like the pruning you are talking about. That sounds more like an issue with your specific provider to me.
- trevyn 7y ago

EthanHeilman 7y ago

Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material became available. All joking aside, I do wonder wither digital or analogue formats are better able to survive into the distant future. * What impact will DRM have on the accessibility of our knowledge to future historians? * Is anything recoverable from a harddrive or flash media after 500 years in a landfill? * Will compressed files be more of less recoverable? What about git archives? * Will the future know the shape of our plastic GI Joes toys but not the content of the GI Joes cartoon?

qmmmur 7y ago
Pretty much everyone in a tech job could afford to buy 40TB of storage at home, or remotely and mirror the entire repo. I think that given this low barrier of entry if you can afford to help preserve the information then you can and probably should. Even if a small amount do it it's more points of recovery.
- int_19h 7y ago
  Storage isn't hard, but downloading 40 Tb can be a problem. Are there any arrangements for physical distribution (of the "truck loaded with USB drives" variety)?
- asdefghyk 7y ago
  Id say.... in the day .... anyone could afford to buy a single floppy disk and store files on it. But how many actually did and how many are actually recoverable. Lots probably got thrown out in intervening years.
unicornporn 7y ago
In the GLAM sector the LOCKSS[1] is project is quite well-known. It tries to deal with some of the resiliency problems that is inherent in digital preservation. However, I'd guess this system does not offer the needed anonymity. [1] https://www.lockss.org/ https://www.lockss.org/ ; https://en.wikipedia.org/wiki/LOCKSS https://en.wikipedia.org/wiki/LOCKSS

nullifidian 7y ago

Posting that here only creates problems for them. The more it's known in the west the more likely it will go down.

coffee12345 7y ago
+1, bookwarrior has warned about this.
- news_hacker 7y ago
  who is bookwarrior?

skjoldr 7y ago

Can't Tahoe-LAFS help with this kind of a challenge? I don't have experience with it, but it looks stable.

knzhou 7y ago

Libgen is one of the greatest contributors to scientific productivity worldwide, possibly beaten only by Sci-Hub. Just about everybody in academia knows about it. If it ever vanished, some of us could probably still get by trading files from person to person, but nothing could be as perfect as what we got now.

lioeters 7y ago
> possibly beaten only by Sci-Hub Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source. So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?). > Just about everybody in academia knows about it. That really says something about the state of society, this tension between copyright laws (and the motivations behind them) and the intellectual ideal of free and open access to knowledge.
- mikorym 7y ago
  I am not an expert on the topic, but I believe that in the former Soviet Union it was common between mathematicians to pass around preprints (a la arXiv). These then perculated through to the West. I think it had to do with the USSR and their restrictive (if we are being euphemistic) policies towards academics.
  - gdy 7y ago
    "the USSR and their restrictive (if we are being euphemistic) policies towards academics." What do you mean?
    - alteria 7y ago
      Their policies were for more than "restrictive" is how I'm reading it See [1] [1] https://en.wikipedia.org/wiki/Suppressed_research_in_the_Soviet_Union https://en.wikipedia.org/wiki/Suppressed_research_in_the_Sov...

boksiora 7y ago

its best to split on small torrents on few 1-2 GB so normal users can seed

milofeynman 7y ago

I'd like to dedicate 1TB of my FreeNAS to something like this. Would be nice to run a small container with some P2P service that contained that chunk.

whydoyoucare 7y ago

Isn't scanning a physical book and uploading a soft-copy, a landmine of hazards (both legal and moral)? Essentially you are encouraging (some) unlawful activity... I am not so sure I am onboard with this idea!

guidoism 7y ago
It’s easy to take this stance in a rich country. But what about the people in countries where one of these books cost the equivalent of a year’s wages. Not so black and white eh?
- whydoyoucare 7y ago
  As far as I know, prices of books differ between rich and developing nations. For e.g., The C Programming Language that costs $50 in the US [1], is sold for Rs. 259 (~$4 US) in India. I believe that is the case with most "economy editions" specifically targeted at developing nations. It certainly isn't an "year's wages". While I do understand your point, it still does not justify encouraging modern-day Robinhoods' and breaking the law. [1] https://www.amazon.com/Programming-Language-2nd-Brian-Kernighan/dp/0131103628 https://www.amazon.com/Programming-Language-2nd-Brian-Kernig... [2] https://www.amazon.in/Programming-Language-Kernighan-Dennis-Ritchie/dp/9332549443/ https://www.amazon.in/Programming-Language-Kernighan-Dennis-...
  - guidoism 7y ago
    Economy editions are nice, but only make up a tiny fraction of what's out there to consume. Again, $4 is great for a relatively rich nation like India, but what about Eritrea? Maybe that was a little flippant, but The Law in a big rich country is pretty meaningless to someone trying to make a better life for themselves in a poor country.
  - esotericn 7y ago
    Hi. You're speaking to someone from the UK. When I grew up we had to walk uphill both ways, to quote the Four Yorkshiremen sketch. I couldn't afford to buy things like books, operating systems, games, etc. We spent the money on food, rent, and on the Internet. People like that past version of me are going to pirate regardless of what you say. Law breaking? You can't see me grinning ear to ear. I like to think my tax bill makes up for that.

sandov 7y ago

Let me say this: I fucking love libgen. It actually makes my life better and I'm so thankful to the people running it.

mutant 7y ago

I'd say IPFS, but That's a pretty big commitment from an entire community to keep alive.

lukebuehler 7y ago

To me, an aspiring scholar, LibGen is the most amazing tool ever. Things like inter-library loan and access to databases on university networks already make life so much easier to what it used to be—but nothing beats LibGen in terms of convenience. I’m in a the nowadays obscure field of patristic theology and I can’t believe how much stuff I can find on LibGen, often things that even highly specialized research libraries like Harvard don’t have. The hours that LibGen saved me in gathering all the sources for my research must be in the hundreds. Thank you!

6510 7y ago

Imagine this: - A tiny well behaved client that starts with the OS. - It downloads rare bits of the archive at 1 kb/s obtaining 1 GB every 278 hours. It should stop around 100 MB to 5 GB. - It periodically announces what chunks/documents it has. - It seeds those chunks at 1 kb/s - Chunks/documents that have thousands of seeds already are not announced. Eventually those are pruned. This escalates the situation to the point where everyone can help without it costing anything. If someone is trying to obtain a 20 mb pfd it would take 5 and a half hours using a single 1 kb seed. With just 50 seeds it's just 8 min.