5 ms·

Current shadow libraries (zlib, libgen, scihub) suffer from centralized data hosting and opaque librarians/custodians (who modify metadata and gate inclusion/ex

by pradn 4y ago

Current shadow libraries (zlib, libgen, scihub) suffer from centralized data hosting and opaque librarians/custodians (who modify metadata and gate inclusion/exclusion of content). We already have the tools to solve this. 1. Files are stored in a distributed fashion and referred to via their content hash. We already have IPFS for this. 2. Library metadata can be packaged up into a SQLite DB file. The DB would contain IPFS hashes, book names, authors, etc. 3. Teams of volunteers assemble and publish the library metadata DB files. There can be multiple teams, each with their own policies. The latest library files can be published via RSS. Each team can have their own upload portal. 4. A desktop app can pull multiple RSS feeds for multiple libraries. The libraries can be combined together and be searched easily on the client side. Users can search for content via the latest library metadata files, locally on their desktop. Content can be downloaded via IPFS. 5. The desktop app can also double as an IPFS host, allowing users to choose specific files to pin or simply allocate an amount of space for the purpose (100 GB, etc). There could also be servers that aggregate pinning info to make sure no gaps are there. 5. For ease of access, people can run websites that preclude the need to setup your own desktop app / download libraries. 6. Library teams can publish metadata DBs and content via torrents, too, for long-term/disaster-recovery/archival purposes. This would be a true hydra. No one centralized team, no reliance on DNS. If one team's library set up goes down, you can use another's.

dark-star 4y ago
1. yes, ipfs could solve that, but it relies on people hoting content. Previous examples of content-based addressing showed that little-accessed content tends to disappear as nodes go offline over the years. This would need to be solved, and I think the only way to solve it is to have a battery of centralized ipfs servers mirroring each other, which defeats the "fully distributed" setup 2. this would also need to be hosted and could be taken down. You'd need to mirror this too, but that's a simpler problem to solve (gigabytes instead of terabytes) 3. the upload portals and the RSS feeds would, again, be centralized or would have to change so regularly that they become impractical in the end you would end up with a dozen (a hundred? more?) different z-libraries, which would make it actually worse from a preservation standpoint, since only the most popular content would be shared, libraries that focused on rare/exotic/fringe material would be endangered of being lost since they have fewer volunteers/mirrors/seeds/... Also, freenet and other projects already showed that end-users allocating some storage and using that to spread data around is not an easy problem, the fluctuation in end-nodes is so big that it slows down the entire network to a crawl. I'm not sure this problem has been solved yet.
- yamtaddle 4y ago
  The core problem with IPFS and friends is that the vast majority of Web access these days occurs on battery-powered devices that need to sleep their connections often and for long periods or battery life will plummet. End users aren't going to accept even 10% worse battery life (and it'd likely be worse than that) just so they can participate in "the swarm" and have higher latency and slower transfer on all their requests.
- pradn 4y ago
  1. You're absolutely right that IPFS alone isn't a good way to guarantee durability. I think you'd need a second level of archiving done in bulk. Thousands of books collated into various torrents that people can help seed. This has already been done for LibGen. IPFS does provide a common hash space for everyone to rally around, and does make it easy to download single books. It's also easy to speed up downloads by using an IPFS gateway - the actual protocol is slow. I don't expect most users to actually pin files or anything. 2. Well, users and librarians need some way to find each other. That's true in any system. And that communication medium can allow certain kinds of attacks. (website on the public internet, word-of-mouth, Telegram groups) If all someone needs is an IPFS hash of a recent library metadata DB (a SQLite file), any way of communication will suffice. I think this approach allows for centralization (sure, keep a website up as long a the authorities don't care) but also gracefully allows for all manner of de-centralization (use any of the above methods to distribute the metadata DB). 3. Any many-to-one system with curation (librarians) will have weak points. The idea is you can set up upload portals across any communication medium (a regular website, a dark-net site, a Telegram group, email) - and the libraries take care of collating the information. The social grouping is what matters more (libraries vs uploaders vs downloaders) - and we want to make it tech agnostic and, therefore, more resilient. This system will be stable I think, for two reasons: 1. Network and branding effects will naturally create a few big libraries. People will use the familiar, useful libraries. See how few torrent search sites took up the bulk of traffic, back in the heyday of torrents. Most users will probably use a website, and the ones that are easiest to use will probably get the most traffic. 2. The resilience of the system is necessary only once in a while. A set of libraries will emerge, there'll be enforcement actions and they might break apart, and then new ones will pick up their pieces (easily bc the metadata and content is open). So we want to provide the open-ness for this to actually happen.
__MatrixMan__ 4y ago
You could keep the metadata DB in IPFS too, along with the books. The only thing that needs to be regularly republished is the CID of the metadata DB, which fits in a tweet. In fact, it fits in a tweet with enough room for steganographic deniability hijinks. You could publish a browser plugin which transforms tweets into IPFS CIDs according to a non-cryptographic hash algorithm. That way the cleartext of your tweet is not takedown-worthy, nor is the plugin, but the two together let users transform the seemingly innocuous tweet into the metadata DB update they need.
- pradn 4y ago
  That's a great idea, and also amusing. :) It's amazing that we can refer to data in a globally-unique way with small content-based hashes. Hash collisions aren't usually worth worrying about. Another benefit is that its easy to store large numbers of hashes with basic metadata. SHA-256 hashes are 32 bytes. If it takes 512 bytes on average to store author/title/publish-date/ISBN, then the hash is a small part of the total per item (though not well-compressible.) You can store the info for 2 million books in a megabyte. Shadow librarians can also publish curated collections of books. I know a guy who tried to do this in a systematic way for college-level history textbooks covering a wide swathe of the world's history. The entire catalog with metadata and hashes is probably only a few hundred thousand KB.
  - __MatrixMan__ 4y ago
    The tricky part is finding a balance between the blockchain approach, where everybody pins every item (wasteful) and the other end of the spectrum where people only pin the things they're thinking about right now (lossy). There's some middle ground where we coordinate who pins what in a way where there's just enough enough redundancy to not worry about it disappearing, but not so much that any one of us is bearing an unnecessarily large burden. I don't think we've quite figured that part out yet.
    - pradn 4y ago
      You bring up a great insight - that blockchain/pinning are on opposite ends of the spectrum of what one can choose to store, if one participates in the system at all. We do have a good example of this already I think: torrent client peer lists. When participating in a torrent, we can see the current availability of the data in the swarm, displayed usually as a bar visualization, with each pixel being a chunk. The darker the chunk, the more peers have a copy of that chunk. The result is also summarized in a number capturing the general health of the swarm wrt hosting the torrent. All we need to do, then, is to have a mechanism where 1) the client has a list of items to pin 2) the client uses this existing swarm-tracking mechanism to figure out which files need more hosts 3) the client picks the best ones to host given the available space/network constraints. One can be smarter than just picking the lowest-seeded file. If a host is known to have files consistently, but is offline for a few hours a day, the client can be smart and not worry about immediately getting those files, perhaps spending available resources on less-seeded files. This is possible with current technology. We can do a simple version of this via a torrent client plugin, reading the list of files from RSS or a special text file. I've seen communities do this manually actually. For hosting SciHub torrents, the community made a web page that showed the current number of known seeds per torrent. Users were responsible for picking whichever ones, usually the lowest-seeded ones, and seeding them. We can remove this tedious and error-prone work. Doing this per IPFS file will probably take up too many resources. Perhaps we need a standard IPFS<->torrent correspondence. Something as simple as a text file in the root of the torrent file structure, a file that maps IPFS hash <-> file inside the torrent. This way an IPFS swarm and a torrent swarm can work together. You get the easy retrieval of IPFS and the increased durability of torrent swarms.
shaunsingh0207 4y ago
https://annas-archive.org https://annas-archive.org does a lot of this, and has been my go-to for books for a while now.
- pradn 4y ago
  Yes! Anna has been active in helping duplicate Zlib, among other efforts. No one person can do it all. We need to pitch in!
ornornor 4y ago
An alternative that works today is #bookz on undernet.
mich_mechanic 4y ago
Guys, you already have this one named Nexus. Even for torrents. https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obofxujdoxa.ipfs.ipfs.joaoleitao.org/#/ https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obof... You must learn about them.
superpirate 4y ago
It has already done, you will be glad to learn: https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obofxujdoxa.ipfs.ipfs.joaoleitao.org/#/ https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obof... For me it has worked in Brave, Chrome, Firefox and Safari Tech Preview, with or without installed IPFS and IPFS Companion (with IPFS it works much better). Haven't worker in Safari in IOS and non-Chrome browsers on Android. Not very fast and sometimes it requires page reloading but overall impression is awesome.
- pradn 4y ago
  Thank you! This looks promising!

by pradn 4y ago

dark-star 4y ago

1. yes, ipfs could solve that, but it relies on people hoting content. Previous examples of content-based addressing showed that little-accessed content tends to disappear as nodes go offline over the years. This would need to be solved, and I think the only way to solve it is to have a battery of centralized ipfs servers mirroring each other, which defeats the "fully distributed" setup 2. this would also need to be hosted and could be taken down. You'd need to mirror this too, but that's a simpler problem to solve (gigabytes instead of terabytes) 3. the upload portals and the RSS feeds would, again, be centralized or would have to change so regularly that they become impractical in the end you would end up with a dozen (a hundred? more?) different z-libraries, which would make it actually worse from a preservation standpoint, since only the most popular content would be shared, libraries that focused on rare/exotic/fringe material would be endangered of being lost since they have fewer volunteers/mirrors/seeds/... Also, freenet and other projects already showed that end-users allocating some storage and using that to spread data around is not an easy problem, the fluctuation in end-nodes is so big that it slows down the entire network to a crawl. I'm not sure this problem has been solved yet.

yamtaddle 4y ago
The core problem with IPFS and friends is that the vast majority of Web access these days occurs on battery-powered devices that need to sleep their connections often and for long periods or battery life will plummet. End users aren't going to accept even 10% worse battery life (and it'd likely be worse than that) just so they can participate in "the swarm" and have higher latency and slower transfer on all their requests.
pradn 4y ago
1. You're absolutely right that IPFS alone isn't a good way to guarantee durability. I think you'd need a second level of archiving done in bulk. Thousands of books collated into various torrents that people can help seed. This has already been done for LibGen. IPFS does provide a common hash space for everyone to rally around, and does make it easy to download single books. It's also easy to speed up downloads by using an IPFS gateway - the actual protocol is slow. I don't expect most users to actually pin files or anything. 2. Well, users and librarians need some way to find each other. That's true in any system. And that communication medium can allow certain kinds of attacks. (website on the public internet, word-of-mouth, Telegram groups) If all someone needs is an IPFS hash of a recent library metadata DB (a SQLite file), any way of communication will suffice. I think this approach allows for centralization (sure, keep a website up as long a the authorities don't care) but also gracefully allows for all manner of de-centralization (use any of the above methods to distribute the metadata DB). 3. Any many-to-one system with curation (librarians) will have weak points. The idea is you can set up upload portals across any communication medium (a regular website, a dark-net site, a Telegram group, email) - and the libraries take care of collating the information. The social grouping is what matters more (libraries vs uploaders vs downloaders) - and we want to make it tech agnostic and, therefore, more resilient. This system will be stable I think, for two reasons: 1. Network and branding effects will naturally create a few big libraries. People will use the familiar, useful libraries. See how few torrent search sites took up the bulk of traffic, back in the heyday of torrents. Most users will probably use a website, and the ones that are easiest to use will probably get the most traffic. 2. The resilience of the system is necessary only once in a while. A set of libraries will emerge, there'll be enforcement actions and they might break apart, and then new ones will pick up their pieces (easily bc the metadata and content is open). So we want to provide the open-ness for this to actually happen.

__MatrixMan__ 4y ago

You could keep the metadata DB in IPFS too, along with the books. The only thing that needs to be regularly republished is the CID of the metadata DB, which fits in a tweet. In fact, it fits in a tweet with enough room for steganographic deniability hijinks. You could publish a browser plugin which transforms tweets into IPFS CIDs according to a non-cryptographic hash algorithm. That way the cleartext of your tweet is not takedown-worthy, nor is the plugin, but the two together let users transform the seemingly innocuous tweet into the metadata DB update they need.

pradn 4y ago
That's a great idea, and also amusing. :) It's amazing that we can refer to data in a globally-unique way with small content-based hashes. Hash collisions aren't usually worth worrying about. Another benefit is that its easy to store large numbers of hashes with basic metadata. SHA-256 hashes are 32 bytes. If it takes 512 bytes on average to store author/title/publish-date/ISBN, then the hash is a small part of the total per item (though not well-compressible.) You can store the info for 2 million books in a megabyte. Shadow librarians can also publish curated collections of books. I know a guy who tried to do this in a systematic way for college-level history textbooks covering a wide swathe of the world's history. The entire catalog with metadata and hashes is probably only a few hundred thousand KB.
- __MatrixMan__ 4y ago
  The tricky part is finding a balance between the blockchain approach, where everybody pins every item (wasteful) and the other end of the spectrum where people only pin the things they're thinking about right now (lossy). There's some middle ground where we coordinate who pins what in a way where there's just enough enough redundancy to not worry about it disappearing, but not so much that any one of us is bearing an unnecessarily large burden. I don't think we've quite figured that part out yet.
  - pradn 4y ago
    You bring up a great insight - that blockchain/pinning are on opposite ends of the spectrum of what one can choose to store, if one participates in the system at all. We do have a good example of this already I think: torrent client peer lists. When participating in a torrent, we can see the current availability of the data in the swarm, displayed usually as a bar visualization, with each pixel being a chunk. The darker the chunk, the more peers have a copy of that chunk. The result is also summarized in a number capturing the general health of the swarm wrt hosting the torrent. All we need to do, then, is to have a mechanism where 1) the client has a list of items to pin 2) the client uses this existing swarm-tracking mechanism to figure out which files need more hosts 3) the client picks the best ones to host given the available space/network constraints. One can be smarter than just picking the lowest-seeded file. If a host is known to have files consistently, but is offline for a few hours a day, the client can be smart and not worry about immediately getting those files, perhaps spending available resources on less-seeded files. This is possible with current technology. We can do a simple version of this via a torrent client plugin, reading the list of files from RSS or a special text file. I've seen communities do this manually actually. For hosting SciHub torrents, the community made a web page that showed the current number of known seeds per torrent. Users were responsible for picking whichever ones, usually the lowest-seeded ones, and seeding them. We can remove this tedious and error-prone work. Doing this per IPFS file will probably take up too many resources. Perhaps we need a standard IPFS<->torrent correspondence. Something as simple as a text file in the root of the torrent file structure, a file that maps IPFS hash <-> file inside the torrent. This way an IPFS swarm and a torrent swarm can work together. You get the easy retrieval of IPFS and the increased durability of torrent swarms.

shaunsingh0207 4y ago

https://annas-archive.org https://annas-archive.org does a lot of this, and has been my go-to for books for a while now.

pradn 4y ago
Yes! Anna has been active in helping duplicate Zlib, among other efforts. No one person can do it all. We need to pitch in!

ornornor 4y ago

An alternative that works today is #bookz on undernet.

mich_mechanic 4y ago

Guys, you already have this one named Nexus. Even for torrents. https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obofxujdoxa.ipfs.ipfs.joaoleitao.org/#/ https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obof... You must learn about them.

superpirate 4y ago

It has already done, you will be glad to learn: https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obofxujdoxa.ipfs.ipfs.joaoleitao.org/#/ https://bafybeigpp6mtsmjngaqscfkjwzivbptt4ui5yb7uih6qe43obof... For me it has worked in Brave, Chrome, Firefox and Safari Tech Preview, with or without installed IPFS and IPFS Companion (with IPFS it works much better). Haven't worker in Safari in IOS and non-Chrome browsers on Android. Not very fast and sometimes it requires page reloading but overall impression is awesome.

pradn 4y ago
Thank you! This looks promising!