22 ms·
Libgen Storage Decentralization on IPFS
- Cactus2018 6y agoWikipedia > The InterPlanetary File System (IPFS) is a protocol and peer-to-peer network for storing and sharing data in a distributed file system. IPFS uses content-addressing to uniquely identify each file in a global namespace connecting all computing devices.
- fsflover 6y agoI think such materials should better be placed to darknets protecting the privacy of users and seeders, e.g., to I2P.
- bboygravity 6y agoI agree. Actually in this day and age I think all information sharing should better be placed on darknets. I don't see the net advantage to anybody of being profiled constantly on clearnet. Naive question: is there such a thing as Bittorrent over I2P or otherwise anonymized decentralized file sharing?
- segfaultbuserr 6y ago> Bittorrent over I2P Yes. I2P is designed with P2P in mind. The default Java I2P client already comes with a bundled BitTorrent client, there is also a BitTorrent client called "Vuze" and is used by some non-anonymous users to help seeding a torrent across the clearnet and I2P simultaneously. Because of the nature of I2P, don't expect great speed. No seeder is "usual", 10 KiB/s is "normal" and 50 KiB/s with only 3 or 5 seeders is "good", overall, it feels as if it's still the early 2000s, but when you get used to it and adopt a "download and forget, come back in 3 days" mentality, it's not actually that bad. The fastest download I've ever seen was the movie Snowden, 20+ seeders and over 200 KiB/s+...
- deleted 6y ago[deleted]
- RealityVoid 6y agoI see it as 2 separate problems: data availability and anonymous transfer. IMO, you need 2 separate solutions because they are 2 different problems. IPFS solved the data availability problem because you can incentivize storage miners to make your data available. This is something new, IMO. Accessing it anonymously is possible with an extra protocol layer.
- apples_oranges 6y agoFor instance, using TOR to download books from libgen should be pretty safe, no?
- LockAndLol 6y agoIt'd be nice for IPFS to support an I2P proxy. That would be quite an important change and would save anybody downloading from LibGen over IPFS.
- derefr 6y agoIf you take IPFS's model and add inherent anonymity, that's exactly what Freenet is. (To this day I'm confused about why IPFS is popular while Freenet isn't.)
- chriswarbo 6y agoI've not tried Freenet for a few years, but every time I did it was very slow; e.g. browsing basic HTML sites hosted on Freenet was quite painful. IPFS can serve streaming video, and host massive archives like Wikipedia. Also, Freenet actively seeks out data from the network to store locally, to give plausible deniability to users (there's no way to tell if something is locally cached because the user requested it, or whether the client fetched it automatically). IIRC this can be disabled by limiting oneself to a whitelist of peers (preventing network chatter that might reveal Freenet's presence). IPFS will only cache what's requested, so it's more predictable and useful for building infrastructure (e.g. hosting assets for a normal Web site); and presumably leaner on data requirements too. IPFS isn't really competing with Freenet IMHO; it's competing with HTTP and BitTorrent.
- eMGm4D0zgUAVXc7 6y ago> I've not tried Freenet for a few years, but every time I did it was very slow It has become a lot faster recently, try again :) When I look at its statistics during active usage it can easily show 1 MiB/s traffic. You still can't expect HTML to load in sub-second delays, for large multi-MiB sites it may take a low single-digit amount of seconds, but there is an inherent cost to anonymity which probably induces a boundary on how fast things can be: To achieve anonymity, data needs to be redirected across multiple people so the sender and recipient can't determine who each other of them is. Redirecting stuff across a longer path than necessary slows things down.
- bawolff 6y ago> To achieve anonymity, data needs to be redirected across multiple people so the sender and recipient can't determine who each other of them is. > > Redirecting stuff across a longer path than necessary slows things down. That's skipping past a lot of details. There are still choices there. Freenet traded speed, efficiency and scalsbility for a specific security model. There are anonoyminity systems (e.g. tor) that are much more efficient but have different trade-offs.
- segfaultbuserr 6y agoThe IPFS mirror really gives a substantial performance boost. In the past, the files had to be fetched from a tired server far away in Russia, the speed can be as low as 0.5 Mbps in my experience (depends on your ISP and connectivity), now it can be lightning fast thanks to P2P, 2 Mbps+, more than adequate to download a 100 MiB+ file. You don't even have to install IPFS (but you probably should, to maintain decentralization), books can be downloaded straight via HTTPS from Cloudflare's IPFS reverse proxy (called a "gateway"), cloudflare-ipfs.com. The IPFS network itself cannot be taken down, but I wonder when will the publishers discover IPFS web gateways, and start filing DMCA takedown requests against them, and what are the gateways gonna do? Will Cloudflare implement a huge copyright blocklist or even terminate its service (or worse, send a list of IP addresses to Prentice Hall or Elsevier)? What about the semi-official ipfs.io gateway? And the smaller independent IPFS gateways (there are currently 10+ on the public web)? Will they be DMCAed out of existence?
- traverseda 6y agoIn my experience IPFS is only fast when you're using it through something like Cloudflare's IPFS proxy (which is basically a caching proxy). I haven't found IPFS to be actually usable any time I've tried running a node myself. Is that unusual?
- segfaultbuserr 6y agoInteresting, didn't notice that before. I believed my downloads are too obscure to be cached by Cloudflare, but I may be wrong. I'll do a comparison once I fixed my server at home...
- the_duke 6y agoNo. The last time I tried it (maybe 9-12 months ago) it was a real resource hog (including choking the network) and really slow.
- magila 6y agoThis has been the state of IPFS for years now. The protocol and client haven't been developed to where they can work reliably in a real P2P scenario. IPFS only appears to work because most requests go strait to one of the big gateways which already has the content cached. Development of IPFS slowed to a crawl when most of Protocol Labs switched to working on Filecoin. It remains to be seen if either Filecoin or IPFS will ever be viable for their intended purpose.
- petra 6y agoWhat's missing from this system(and P2P systems in general), is automatic pinning - i.e. the system automatically choses which files to pin on which user, to ensure every file is available on the net. It's a bit of tricky problem, because people may not like unknown, random files on their pc, and probably, because such algorithm is distributed and will need to resist attacks. But i think that the payoff is big: what Bitorrent did to popular files, such an algorithm could do to rare files.
- crest 6y agohttps://cluster.ipfs.io/ https://cluster.ipfs.io/
- rudolph9 6y agoThis doesn’t automatically pin, rather distributes a pin set across a set of trusted nodes. It’s good stuff nonetheless.
- rudolph9 6y agoIpfs has something called “bitswap” which solves some of what you’re describing but is lacking in areas such as long term file availability. That’s where filecoin sort of finds sort fills in the gap. https://docs.ipfs.io/concepts/bitswap/ https://docs.ipfs.io/concepts/bitswap/
- corey_moncure 6y agoIf the pinning came with encryption where the key is not known to the pinned system, I think most would be OK with it. There's no way to know, or demonstrate that anyone could know, the contents of any given file. Furthermore, the file could be broken into encrypted chunks and distributed redundantly to many users. Kind of like BT turned inside out.
- derefr 6y agohttps://en.wikipedia.org/wiki/Perfect_Dark_(P2P) https://en.wikipedia.org/wiki/Perfect_Dark_(P2P) is this kind of system. You allocate a slice of your disk for the program to use as a global cache, and random encrypted chunks from the network end up downloaded into it.
- Vespasian 6y agoI'm hesitant to participate in IPFS in the context of libgen. Torrenting copyrighted materials (especially if you are distributing) can be quite expensive in some jurisdictions and I do not believe that IPFS is treated different than regular BitTorrent.
- whimsicalism 6y ago> Torrenting copyrighted materials (especially if you are distributing) can be quite expensive in some jurisdictions Why? Your ISP charges for BitTorrent traffic?
- dewey 6y agoIn case you are serious: Lawyer agencies are joining the swarm, recording IPs, write letters to ISPs and send you fines. That's very common for example in Germany. Edit: I didn't downvote you if it looks like that :)
- whimsicalism 6y ago> write letters to ISPs and send you fines. That's very common for example in Germany. And you have to pay the fines? That's nuts. In the US, you are pretty much as likely to get fined for torrenting as you are to get arrested for jaywalking (although without the racial disparity).
- dewey 6y ago> And you have to pay the fines? That's nuts. Yes, conveniently there's a second group of lawyers that focuses on getting people out of these fines...for a fee that's not so much lower. Unfortunately it's a bit of a business model in some countries.
- nemo 6y agoIn the US the (very) common scenario is to get a couple warnings from your ISP and if you continue torrenting copyrighted material you will be dropped as a customer and banned.
- MichaelMoser123 6y agodoes anyone know why libgen is still running over http? (But sincerely thanks for the site! it saved me on several occasions!)
- warmwaffles 6y agoProbably because they don't care enough to add https.
- generationP 6y agoThey don't want the extra weak spot of certificate revocation being abused against them? Though I'm not sure how big of an issue this is; sci-hub is using https.
- jdu9 6y agoUnless they activate HSTS, it shouldn't be a problem. They can always go back to HTTP if they want. Also, Let's Encrypt doesn't even revoke certificates for malware and phishing sites. [1] Maybe they're doing this for better caching performance? It makes SSL flood attacks impossible, but is that really worth it? [1] https://community.letsencrypt.org/t/how-to-report-abuse/41106/4 https://community.letsencrypt.org/t/how-to-report-abuse/4110...
- dublinben 6y agoOne of the new IPFS-backed mirror sites (https://libgen.fun/ https://libgen.fun/) is on HTTPS.
- MichaelMoser123 6y agoGreat, however it currently doesnt have the sci-tech filter, and no comics filter too :-(
- givepause 6y agoscitech is there, it's relabeled "Educational." comics is not a public dataset -- it's privately run and managed, so there are no mirrors for it. I guess someone could scrape it.
- dooglius 6y agoA strange choice given that distributing copyrighted materials violates the IPFS Code of Conduct [0]. Why not use BitTorrent whose attitude toward copyright is much more in sync with libgen's? [0] https://github.com/ipfs/community/blob/master/code-of-conduct.md#copyright-violations https://github.com/ipfs/community/blob/master/code-of-conduc...
- segfaultbuserr 6y agoLibGen already has "torrent per 1000 files" for a long time, some books also have their individual torrents but it's not common. I'm not sure why, but I think the problem is that files on BitTorrent cannot be directly addressed by hashes, it had to go through an intermediate step called a "torrent", which does not uniquely identify individual files directly. You can always select the one file you need from a huge torrent, but it's not as convenient as directly addressing files by their hashes.
- dooglius 6y agoThat's fair, BitTorrent v2 does solve this problem, but I can't say I've seen any v2 torrents around (not that I'm specifically looking)
- elvis70 6y agoAh, thanks for mentioning that. If I understand well, it's a distributed filesystem but there is still an authority able to ban users or contents?
- donpdonp 6y agoThe linked article does not introduce the project, nor does the project site have an about page. wikipedia to the rescue! https://en.wikipedia.org/wiki/Library_Genesis https://en.wikipedia.org/wiki/Library_Genesis
- frrad 6y agoI wrote this https://github.com/frrad/skyhub https://github.com/frrad/skyhub the other day to provide a nicer interface on top of the scihub torrents.
- fareesh 6y agoIs book piracy a major concern for the publishing industry? Many avid readers that I know are happier buying the physical book because they like the feel of reading from a book. Have others found this to be true as well? I wonder if younger folks are less susceptible to this feeling.
- Synaesthesia 6y agoMost people don't know how to pirate books but yeah it is. I got a lot of university textbooks this way.
- contravariant 6y agoMany avid readers I know are happy with physical books but also happy to use an e-reader for purely pragmatic reasons (easier to carry, custom font settings). As for piracy if the places I see libgen and it's kind referenced are any indication then this isn't about just any readers but about students, professors etc. who want to access scientific publications.
- justarandomq 6y agoIs LibGen safe? I remember testing some of its mirror links on virustotal and saw some red flags, so I decided not to risk downloading.
- Synaesthesia 6y agoYes it's just ebooks.
- mpfundstein 6y agoand scientific papers
- CamperBob2 6y agoEven an ordinary .PDF can have executable content. This isn't a baseless concern.
- mpfundstein 6y agowhat red flags did you see?
- justarandomq 6y agoIt was some book, and when submitting the download link, 3 of the antivirus results were red. Perhaps they were false positives.
- lucb1e 6y agoAnti-Virus tools have been flagging keygens and cracks for ages, and also hacking tools that grab passwords from (system) stores even if the tools have absolutely nothing in their code that talks to a server (I only ever used those for legal purposes, but can't tell you how often I had to disable an annoying AV...). Just because the AV says something is malware doesn't mean you can trust its claim. From my perspective, they've undermined their trustworthiness in the area where you are indeed most likely to rely on an AV. I'm not surprised they're now flagging what is basically text and images as malware.
- kristianpaul 6y ago"- Recommended at least 16GB RAM and Intel i5 or equivalent processor - Recommended at least 100 mbps, gigabit connection preferred" And I was going to try this one of my embedded computers :(
- xvilka 6y agoBy the way, there is also a LibGen plugin[1] for Calibre book manager and reader. [1] https://github.com/MCOfficer/LibGen-calibre https://github.com/MCOfficer/LibGen-calibre
- Proven 6y agoSuch use case! Can this solution in search of a problem find any real problem to solve?