9 ms·

Putting Z-Library on IPFS

356 points by NKosmatos 4y ago

typingmonkey 4y ago
You cannot "put stuff" on ipfs. Either you seed it or someone else does. If noone seeds it, it is gone. I bet most of it will be offline in 3 months.
- tshaddox 4y ago
  That’s clear from the article, which is about the nontrivial task of making tens of terabytes of data available on IPFS.
- Xeoncross 4y ago
  Isn't that true of anything shared on a network?
  - acdha 4y ago
    Yes, but there are people who claim that IPFS is uncensorable or permanent. I saw that a lot from NFT salespeople who were claiming you’d own something forever with no further payments needed, or even that a copyright claim couldn’t take it down.
- tinalumfoil 4y ago
  This is very nit picky. The entire web works by severs “seeding” data to the client, so by your logic you can’t “put stuff” on the web. The difference with torrents and IPFS is you can have multiple servers seeding the same content, and not be dependent on any one. I’m not making any predictions about how long Z library stays up, but the illegal seeding of movies and tv shows has remained very strong until today.
  - typingmonkey 4y ago
    Fair point. From my experience people often think ipfs works equal to sth like a shared drive or filecoin, so I had to point that out.
  - dale_glass 4y ago
    IPFS is actually worse to put stuff in than the web or torrents, in my experience. Last time I tried it, the ipfs service used its own storage scheme. Meaning it's not like pointing Apache at a directory. You take your stuff, and upload it into ipfsd first, and it puts that data into its storage system. So to do this from scratch (not mirroring somebody else's content) needs a minimum of >62TB -- 31TB of content, which ipfsd will then package into 31TB more + overhead in its storage area. And of course if you're doing this, you're expecting other people to mirror this stuff, so count on hundreds of terabytes of traffic. So this is easily ~$3K in hard disks alone, plus the NAS/server hardware, plus traffic, plus the willingness to risk the FBI coming and grabbing all of it.
    - aabbcc1241 4y ago
      I know the pain of having to store the payload multiple times when using system like ipfs and zeronet. It maybe helpful to leave the files in the file system as is, and store the metadata in a sqlite database. Then integrate it with a p2p network layer for crowd seeding and even content discovery.
    - boramalper 4y ago
      IPFS isn't my favourite tool either but you are wrong. > So to do this from scratch (not mirroring somebody else's content) needs a minimum of >62TB -- 31TB of content, which ipfsd will then package into 31TB more + overhead in its storage area. IPFS has `nocopy` option for quite some time now, which avoids copying. > And of course if you're doing this, you're expecting other people to mirror this stuff, so count on hundreds of terabytes of traffic. Of course you are expecting other people to mirror this stuff, and naturally that will generate some traffic. How is this not a problem with web mirrors or torrents?
      - dale_glass 4y ago
        > IPFS has `nocopy` option for quite some time now, which avoids copying. Oh, didn't find about that one. Thanks! > Of course you are expecting other people to mirror this stuff, and naturally that will generate some traffic. How is this not a problem with web mirrors or torrents? I mean, if I put a book archive on the web, I'd expect the vast majority of people to just grab whichever book they were interested in. Mirroring is a possibility, but a non-trivial thing to accomplish, and can be discouraged. Meanwhile, on IPFS I'd expect a much higher likelihood of somebody trying to replicate the whole archive, so one would do well to keep that in mind and to be prepared for it.
- blooalien 4y ago
  > … "If noone seeds it, it is gone." … This is a very good argument for primarily using ipfs to host high quality content that people will want to preserve long-term…
- yieldcrv 4y ago
  people are pinning stuff on ipfs with filecoin there is a free service that does this for you till like 5 gigabits, it pins with filecoin https://web3.storage/pricing/ https://web3.storage/pricing/ I have entire sites hosted for free this way, whereas other hosts like vercel and netlify will charge you for traffic. you can just put your big assets on ipfs+filecoin pins and have unlimited traffic. the ipfs CDNs help with performance.
  - miohtama 4y ago
    Interesting. How does the CDN work as web browsers are not yet natively supporting IPFS? Is someone operating the CDN?
    - yieldcrv 4y ago
      The gateways are https, choose one that caches on its own, like cloudflare’s
- miohtama 4y ago
  IPFS is a caching or distributing layer, not a persistent storage. There are some attempts to make a storage that backs IPFS by Filecoin team (Filecoin and IPFS teams have a lot of overlap.)
Gigachad 4y ago
I’ve been interested in IPFS for a while now but I’m still unable to come up with a good explanation of how it’s different to regular torrents. Does anyone have any examples of stuff you can do with IPFS that you can’t with torrents? Seems like over the last 5 years they haven’t really done anything but add crypto buzzwords to the project site.
- rudolph9 4y ago
  The content is the address, no central issuer of the tracker file needed just the CID content ID. Also since the CID is a hash of the data it can be used to validate the data :wink:
  - nodja 4y ago
    That is exactly how bittorrent works as well tho. Bittorrent hasn't needed any trackers for public torrents since 2005 and can work solely off magnet links using the DHT network. Just like how IPFS does it.
    - yellowsir 4y ago
      the difference is, when you have a lot of (small) files. With ipfs you have one global peer pool, with torrents you need a pool per file.
  - teraflop 4y ago
    That's basically the same as torrents with magnet links. As other commenters pointed out, the biggest real difference is that IPFS objects can cross-reference each other by hash, which allows you to do partial updates.
    - layer8 4y ago
      An “update” means a new CID though, and this basically works like a persistent data structure [0]? [0] https://en.wikipedia.org/wiki/Persistent_data_structure https://en.wikipedia.org/wiki/Persistent_data_structure
deleted 4y ago
[deleted]
imhoguy 4y ago
This post really starts to show the right direction: Z-Library - "the desktop app", with built in Tor (for seeders safety), IPFS (for p2p distribution), IPNS (to download updated indexes), with local search engine (no SPOF & convenience), optional at-rest file encryption, and some random pining algorithm to let users donate 1-10GB of local disk space to host random chunks of the library. It needs to be dead easy to let anyone use and contribute.
- RobotToaster 4y ago
  Since it needs to be on all the time, a raspberry pi image maybe? Literally just configure your wifi password, burn the image, and let it rip.
  - bozhark 4y ago
    If I could tack it on to the current network with a single pi+Ethernet connection it would be worthwhile
  - gizmo385 4y ago
    Requiring a raspberry pi and burning an image is not generally accessible to a non technical person imo
    - tfsh 4y ago
      If you're not slightly technically inclined you wouldn't be interested in contributing to Z-Libs decentralised hosting so this isn't an issue.
      - dendriti 4y ago
        If you're limiting support of Z-Libs to only those 'slightly technically inclined', you're leaving a lot of support on the table.
        joshspankit 4y ago
roenxi 4y ago
Between the IPFS & the XMR address at the bottom of the blob post - this is using important technology that didn't exist a few years ago - this blog post wasn't technically feasible in 2012. It has become substantially cheaper to do this sort of crowdfunded guerrilla knowledge sharing.
gw67 4y ago
How to search for a book? Is there a search engine?
- flotzam 4y ago
  https://en.wikipedia.org/wiki/Anna's_Archive https://en.wikipedia.org/wiki/Anna's_Archive
halpmeh 4y ago
Isn’t IPFS a terrible technology to host copyrighted material on? As far as I know, it has no privacy features, so you can see exactly who is hosting which content.
- enriquto 4y ago
  I hope we reach an "I'm Spartacus!" situation here, were none of us can be realistically accused of anything.
  - dale_glass 4y ago
    As far as I know, there's no plausible deniability on IPFS. If you decide to help host some piece of content, you personally have to make the choice to mirror it. Which makes you liable.
    - gmuslera 4y ago
      It is a particular piece of content or a group category of content? You may not be liable if you are hosting The Illiad or some other public domain work, how much you (can?) choose or be aware of which pieces of contents you are hosting, if we are not talking about the whole collection?
      - dale_glass 4y ago
        Yeah, you can pin specific pieces of content, and pin recursively. But what's the point in doing this for legal content? You can just get the Illiad from the Gutenberg Project.
        gmuslera 4y ago
        Every year there are books that end into public domain, either because they were published long enough or they were released in that way recently. Are every one of them in the Gutenberg project? Some of those book even are in their own github project, but it would be nice to have them somewhere, and maybe in a way that could resist censorship o blocking. Or other things than books, like projects or research released in public that some commercial platform centralizes and put a price tag to access them.
wrs 4y ago
Considering the goals of IPFS, I’m surprised it has trouble at this scale. I mean, it’s supposed to be “interplanetary” but this data would fit on two hard drives. The post talks about performance problems with advertising “hundreds” of content IDs. Is it a design problem or just an early implementation problem?
- GauntletWizard 4y ago
  Design. IPFS is a dead end in design because it's properties are designed with statistical perfection to it's anonymity. This can only ever be achieved statistically, because there's always the attack of "Start Removing nodes at random, and when the content disappears when you remove a node that was your one" This is a provably unsolvable problem, but it's okay: nobody actually has this problem but the worst social outcasts. In some places that's itself a problem: what "The worst social outcasts" is varies from place to place, but running IPFS is enough proof for a death sentence in those places.
  - dale_glass 4y ago
    You might be confusing IPFS with something else, like Freenet. As far as I know, there's no such statistical anonymity in it.
    - csande17 4y ago
      Yes, it's fairly easy for attackers to discover the IP address of the people currently hosting a particular piece of content on IPFS: https://docs.ipfs.tech/concepts/privacy-and-encryption/#what-s-public-on-ipfs https://docs.ipfs.tech/concepts/privacy-and-encryption/#what... I think the parent comment is asserting that even if you run IPFS over Tor or something, or even if they try and make IPFS more anonymous in the future, it will always still be vulnerable to the "cut off your Internet and see which node in the network goes down" attack.
- ReaLNero 4y ago
  Decentralization is expensive e.g. downloading a Google Drive file is immediate while a torrent takes some time initialize connections, or the PoW protocol for cryptocurrencies. Perhaps we could narrow down the purpose of IPFS to storing a small amount of data in a database that is impossible to delete and difficult to censor?
russellbeattie 4y ago
A few months ago I decided on a whim that I should download and de-drm my Audible audiobooks. I have a lot of them from a decade+ of monthly credits being used. I hadn't really considered how much space it would take up so I just set off the download process and did other stuff. 650 audiobooks took up nearly 2TB of space before I stopped it, only noticing when my Mac gave me a warning about not having enough space left on the drive. Whoops. (Audible keeps track of your total listening time... It adds up to over 9 months solid. Crazy, but I can think of worse ways to spend that time.) Anyways, if 6 million ebooks takes up 31TB, I wonder how much space the equivalent amount of audiobooks would use? And how much space a decent library of movies has? I'm kinda thinking that guy on reddit who somehow got a hold of a used Netflix edge server was on to something.
- getcrunk 4y ago
  the only way i could get that number to work is : 650 books x 15hrs x 3600 seconds x 320kbit/s divided by 8 bits/byte = 1.4tb id say change the audio format and cut the bit-rate roughly in half. but welcome to data hoarding, start with a 8tb external wd dirve!
  - russellbeattie 4y ago
    Hmm... well, I do tend to try to maximize the amount of listening I get per credit, so I have lots of 20+ hour books. Stephen Fry's reading of the complete works of Sherlock Holmes is nearly 63 hours [1] and that's just a "single" audiobook (and highly recommended). Lots of Great Courses recordings are similar. Rise and Fall of the Third Reich is 57 hours, etc. Also, there's a ton of metadata like images and PDFs, though I'm not sure how much that takes up. And in fact (just checking again) I'm at only 638 books, so it was probably more like 625 books when I did this. Apparently it adds up. 1. https://www.audible.com/pd/Sherlock-Holmes-Audiobook/B06WLMWF2S https://www.audible.com/pd/Sherlock-Holmes-Audiobook/B06WLMW...
  - summm 4y ago
    Audiobooks are usually mono, so 700 kbit/s for uncompressed CD quality. To me parent's numbers sounds like uncompressed. I suspect something went wrong with de-drm/decoding. Should be more like 48 kbit/s for basically transparent mono voice with opus. At this rate, 9 solid months will fit into 142 GB...
yonatron 4y ago
I haven't seen one commenter here concerned with copyright violation or piracy. You're ok with that? Have you never created content of your own, made of your own blood, sweat and tears, and then had it pirated and given away free? Do you condone this? Is this not thievery in your eyes?
- RobotToaster 4y ago
  >You're ok with that? Yes
- getcrunk 4y ago
  the problem is that, in general that content made of "blood,sweat and tears" is already being pillaged by the robber barons who take 90%+ of the profit from your work, at least in the fields of movies, tv, music and books. Not saying anything is right or wrong. just pointing out this complicating factor.
  - timmytokyo 4y ago
    So by this logic, it's ok to further immiserate the poor author since she was already 90% immiserated by the big bad publisher?
  - acdha 4y ago
    There’s a serious citation needed on those numbers but even if they are accurate, theseauthors are not slaves. They voluntarily sign contracts rather than give their work away, self-publish, or stop writing. It seems rather self-serving to argue that you are better equipped to make that decision for them and that means you should get something you want without paying for it.
- deleted 4y ago
  [deleted]
- scotty79 4y ago
  Yes. How about you? How hard was your life destroyed by book piracy (or any other kind)?
- 4y ago
QuadrupleA 4y ago
> It was sad day for the free flow of information, knowledge, and culture. Quick counterpoint to this type of revolutionary rhetoric - writing a book (a good one) takes a lot of time, patience, and hard work. Making it available for free, against the author's wishes, while very easy in the digital age and obviously great for readers, is robbing from the author (if they're still alive). It devalues the craft of writing and makes it even more un-viable as a profession. This may be inevitable, and it has mostly happened with music, journalism, etc., but by putting energy into this kind of project you're only furthering its death along, and perhaps furthering the death of society and culture a little by taking away a lot of the incentive to write anything of value. Pirate away if you want, but thinking this is some kind of virtuous revolutionary act is narcissistic BS.
- js8 4y ago
  I think you should read https://lessig.org/product/free-culture/ https://lessig.org/product/free-culture/, maybe you will change your mind.
  - criddell 4y ago
    Do you see any irony that you are pointing to what is basically an ad for a non-free book?
    - hbbio 4y ago
      On March 25, 2004, Professor Lessig released a PDF version of this book under a Creative Commons license. This HTML version, produced by José Menéndez, is released under the same license. https://www.ibiblio.org/ebooks/Lessig/Free_Culture/Free%20Culture.htm https://www.ibiblio.org/ebooks/Lessig/Free_Culture/Free%20Cu...
  - dudul 4y ago
    I'm surprised I need to pay to read this book :)
- enriquto 4y ago
  Just don't read Kafka then.
js8 4y ago
I wish there was some "library request" mechanism for shadow libraries. I don't mind sharing torrents, especially for older and rare works (which are unlikely to be targeted for copyright), but I can't keep all of them active at the same time, lot of them I have on backup disks. I imagine the client could request for data to become available, then server (person managing a part of shadow library) could make somehow available the files after a certain period, and the client would automatically download the available files.
29athrowaway 4y ago
And this is exactly why I will never use IPFS. I don't want anything to do with anything illegal.
- mdp2021 4y ago
  Checked https://www.ipfs.tech/ https://www.ipfs.tech/ , it seems like a progressive project - "those things that do something good". Unless you mean that you do not want to have anything to do with hammers and screwdrivers.
  - 29athrowaway 4y ago
    Gift cards are a great idea. But who are the #1 users of gift cards right now? Scammers who found in gift cards a good way to conceal their identity and financial info. Today, thousands of elderly people will be scammed by idiots that spend their entire day preying on older people. Elderly people that will lose everything and end up homeless. Fuck the current implementation of gift cards. They cause more problems than they fix. We can't have nice things because there are people out there who are fucking disgusting. Therefore we need accountability built into each technology. Why? because there's a lot of people that fucking suck that's why. Build a simple application that allows people to share plain text and figure out what happens next. Harassment, death threats, organized crime, and all sorts of horrible things. Technocrats and tech enthusiasts are often too idealistic to think about the real world consequences of technology. The person that was having fun implementing convolutional neural networks and decided to publish their research for the benefit of the world would never have imagined that years later the chinese would take that research and build a mass surveillance system that tracked the entire population in real time enslaving billions of people. The people that built Tor would never have imagined the caliber of fucked up shit that now takes place there. And the same with every technology.
    - mdp2021 4y ago
      > gift cards ... accountability built into the technology It depends on what you mean with that «accountability built into the technology». Related case, as you are also speaking of cards: this friend of mine wanted to buy a foreign book (less available in bookstores, where you pay cash). So he purchased an anonymous pre-paid credit card. He found out that during the mess of the past three years, somebody has made it very difficult to keep those cards anonymous: they now request a telephone number to activate them, and in regions where, "coincidentally", telephone numbers are to be associated to legal identities. Now: as you should know, you can bet your life that if our transactions were recorded, we would strictly avoid commerce. (Just like, pretty similarly, we would strictly avoid a non anonymous Internet.) So: your call for accountability should work within the boundaries of what is acceptable. > too idealistic Don't worry, we grow up, and some of us restrain production - exactly in the awareness of what may happen.
dale_glass 4y ago
I'm fairly sure this isn't really safe or really reliable. To my knowledge, IPFS isn't really private, in that both the nodes hosting content can be easily known, and the users requesting content can be monitored. This is bad news for something law enforcement has already taken a serious interest in. IPFS also requires "pinning", which means that unless other people decide to dedicate a few TB to this out of their own initiative, what we have currently is a single machine providing data through an obscure mechanism. If this machine is taken down, the content goes with it. The amount of people that have 31 TB worth of spare storage, care about this particular issue, and are willing to get into legal trouble for it (or at least anger their ISP/host) is probably not terribly large. The work could be split up, but then there needs to be some sort of coordination to somehow divide up hosting the archive among a group of volunteers.
- armchairhacker 4y ago
  > To my knowledge, IPFS isn't really private, in that both the nodes hosting content can be easily known, and the users requesting content can be monitored. This is bad news for something law enforcement has already taken a serious interest in. You can access the data through a VPN. If necessary, the hosts can also encrypt the filenames and data so that, until law enforcement gets the encryption key, they can't know who accesses what (public key and other necessary info would be communicated through Signal). Rotate the filenames so so when one is discovered, past requests can't be tracked. Maybe there is a way to slightly break the protocol to further hide the requests. > IPFS also requires "pinning", which means that unless other people decide to dedicate a few TB to this out of their own initiative, what we have currently is a single machine providing data through an obscure mechanism. If this machine is taken down, the content goes with it. Do you have to explicitly choose what data to pin? If so then this is an issue. If not, and you just pin random chunks, then if we normalize people using IPFS and distributing legal data this will be solved. If we normalize it enough, there will be too many people hosting and using IPFS for law enforcement to reasonably take down. Or we could just have enough activists that are willing to risk being fined or arrested. --- That being said though, I'm still not convinced on IPFS because it seems like it cannot handle much and is excessively inefficient (case in point: this article). The authors of IPFS should release a new protocol which addresses issues like the article's, hopefully before too much adoption.
tkiolp4 4y ago
I love this but I don’t understand one thing. It’s relatively easy to find “Anna’s Archive” on the internet, and although it doesn’t host copyrighted content itself, it does link to websites that do host copyrighted PDFs, epubs, etc. What’s stopping the FBI (or others) from taking down such linked websites? Isn’t just a matter of time until all of them are taken down and the owners sued?
TheDesolate0 4y ago
Sugimot0 4y ago
What advantage does ipfs hold over tor or i2p? it seems like ipfs is a poor fit for this use case, but I'm not very familiar with these things.
boredhedgehog 4y ago
I would have expected there to be some more effort to curate and cull the collection before rehosting it in full size. Z-Library's automatic deduplication was at best perfunctory; too many works were featured in various formats and editions with no indication of their merits or distinctions. I wouldn't be surprised if the file size could be cut in half without any valuable information lost.
luuuzeta 4y ago
Funnily enough Z-Library made it easy for me to buy physical copies of books I wouldn't have bought otherwise.

356 points by NKosmatos 4y ago

You cannot "put stuff" on ipfs. Either you seed it or someone else does. If noone seeds it, it is gone. I bet most of it will be offline in 3 months.

tshaddox 4y ago
That’s clear from the article, which is about the nontrivial task of making tens of terabytes of data available on IPFS.
Xeoncross 4y ago
Isn't that true of anything shared on a network?
- acdha 4y ago
  Yes, but there are people who claim that IPFS is uncensorable or permanent. I saw that a lot from NFT salespeople who were claiming you’d own something forever with no further payments needed, or even that a copyright claim couldn’t take it down.
tinalumfoil 4y ago
This is very nit picky. The entire web works by severs “seeding” data to the client, so by your logic you can’t “put stuff” on the web. The difference with torrents and IPFS is you can have multiple servers seeding the same content, and not be dependent on any one. I’m not making any predictions about how long Z library stays up, but the illegal seeding of movies and tv shows has remained very strong until today.
- typingmonkey 4y ago
  Fair point. From my experience people often think ipfs works equal to sth like a shared drive or filecoin, so I had to point that out.
- dale_glass 4y ago
  IPFS is actually worse to put stuff in than the web or torrents, in my experience. Last time I tried it, the ipfs service used its own storage scheme. Meaning it's not like pointing Apache at a directory. You take your stuff, and upload it into ipfsd first, and it puts that data into its storage system. So to do this from scratch (not mirroring somebody else's content) needs a minimum of >62TB -- 31TB of content, which ipfsd will then package into 31TB more + overhead in its storage area. And of course if you're doing this, you're expecting other people to mirror this stuff, so count on hundreds of terabytes of traffic. So this is easily ~$3K in hard disks alone, plus the NAS/server hardware, plus traffic, plus the willingness to risk the FBI coming and grabbing all of it.
  - aabbcc1241 4y ago
    I know the pain of having to store the payload multiple times when using system like ipfs and zeronet. It maybe helpful to leave the files in the file system as is, and store the metadata in a sqlite database. Then integrate it with a p2p network layer for crowd seeding and even content discovery.
  - boramalper 4y ago
    IPFS isn't my favourite tool either but you are wrong. > So to do this from scratch (not mirroring somebody else's content) needs a minimum of >62TB -- 31TB of content, which ipfsd will then package into 31TB more + overhead in its storage area. IPFS has `nocopy` option for quite some time now, which avoids copying. > And of course if you're doing this, you're expecting other people to mirror this stuff, so count on hundreds of terabytes of traffic. Of course you are expecting other people to mirror this stuff, and naturally that will generate some traffic. How is this not a problem with web mirrors or torrents?
    - dale_glass 4y ago
      > IPFS has `nocopy` option for quite some time now, which avoids copying. Oh, didn't find about that one. Thanks! > Of course you are expecting other people to mirror this stuff, and naturally that will generate some traffic. How is this not a problem with web mirrors or torrents? I mean, if I put a book archive on the web, I'd expect the vast majority of people to just grab whichever book they were interested in. Mirroring is a possibility, but a non-trivial thing to accomplish, and can be discouraged. Meanwhile, on IPFS I'd expect a much higher likelihood of somebody trying to replicate the whole archive, so one would do well to keep that in mind and to be prepared for it.
blooalien 4y ago
> … "If noone seeds it, it is gone." … This is a very good argument for primarily using ipfs to host high quality content that people will want to preserve long-term…
yieldcrv 4y ago
people are pinning stuff on ipfs with filecoin there is a free service that does this for you till like 5 gigabits, it pins with filecoin https://web3.storage/pricing/ https://web3.storage/pricing/ I have entire sites hosted for free this way, whereas other hosts like vercel and netlify will charge you for traffic. you can just put your big assets on ipfs+filecoin pins and have unlimited traffic. the ipfs CDNs help with performance.
- miohtama 4y ago
  Interesting. How does the CDN work as web browsers are not yet natively supporting IPFS? Is someone operating the CDN?
  - yieldcrv 4y ago
    The gateways are https, choose one that caches on its own, like cloudflare’s
miohtama 4y ago
IPFS is a caching or distributing layer, not a persistent storage. There are some attempts to make a storage that backs IPFS by Filecoin team (Filecoin and IPFS teams have a lot of overlap.)

Gigachad 4y ago

I’ve been interested in IPFS for a while now but I’m still unable to come up with a good explanation of how it’s different to regular torrents. Does anyone have any examples of stuff you can do with IPFS that you can’t with torrents? Seems like over the last 5 years they haven’t really done anything but add crypto buzzwords to the project site.

rudolph9 4y ago
The content is the address, no central issuer of the tracker file needed just the CID content ID. Also since the CID is a hash of the data it can be used to validate the data :wink:
- nodja 4y ago
  That is exactly how bittorrent works as well tho. Bittorrent hasn't needed any trackers for public torrents since 2005 and can work solely off magnet links using the DHT network. Just like how IPFS does it.
  - yellowsir 4y ago
    the difference is, when you have a lot of (small) files. With ipfs you have one global peer pool, with torrents you need a pool per file.
- teraflop 4y ago
  That's basically the same as torrents with magnet links. As other commenters pointed out, the biggest real difference is that IPFS objects can cross-reference each other by hash, which allows you to do partial updates.
  - layer8 4y ago
    An “update” means a new CID though, and this basically works like a persistent data structure [0]? [0] https://en.wikipedia.org/wiki/Persistent_data_structure https://en.wikipedia.org/wiki/Persistent_data_structure

deleted 4y ago

[deleted]

imhoguy 4y ago

This post really starts to show the right direction: Z-Library - "the desktop app", with built in Tor (for seeders safety), IPFS (for p2p distribution), IPNS (to download updated indexes), with local search engine (no SPOF & convenience), optional at-rest file encryption, and some random pining algorithm to let users donate 1-10GB of local disk space to host random chunks of the library. It needs to be dead easy to let anyone use and contribute.

RobotToaster 4y ago
Since it needs to be on all the time, a raspberry pi image maybe? Literally just configure your wifi password, burn the image, and let it rip.
- bozhark 4y ago
  If I could tack it on to the current network with a single pi+Ethernet connection it would be worthwhile
- gizmo385 4y ago
  Requiring a raspberry pi and burning an image is not generally accessible to a non technical person imo
  - tfsh 4y ago
    If you're not slightly technically inclined you wouldn't be interested in contributing to Z-Libs decentralised hosting so this isn't an issue.
    - dendriti 4y ago
      If you're limiting support of Z-Libs to only those 'slightly technically inclined', you're leaving a lot of support on the table.
      - joshspankit 4y ago

roenxi 4y ago

Between the IPFS & the XMR address at the bottom of the blob post - this is using important technology that didn't exist a few years ago - this blog post wasn't technically feasible in 2012. It has become substantially cheaper to do this sort of crowdfunded guerrilla knowledge sharing.

gw67 4y ago

How to search for a book? Is there a search engine?

flotzam 4y ago
https://en.wikipedia.org/wiki/Anna's_Archive https://en.wikipedia.org/wiki/Anna's_Archive

halpmeh 4y ago

Isn’t IPFS a terrible technology to host copyrighted material on? As far as I know, it has no privacy features, so you can see exactly who is hosting which content.

enriquto 4y ago
I hope we reach an "I'm Spartacus!" situation here, were none of us can be realistically accused of anything.
- dale_glass 4y ago
  As far as I know, there's no plausible deniability on IPFS. If you decide to help host some piece of content, you personally have to make the choice to mirror it. Which makes you liable.
  - gmuslera 4y ago
    It is a particular piece of content or a group category of content? You may not be liable if you are hosting The Illiad or some other public domain work, how much you (can?) choose or be aware of which pieces of contents you are hosting, if we are not talking about the whole collection?
    - dale_glass 4y ago
      Yeah, you can pin specific pieces of content, and pin recursively. But what's the point in doing this for legal content? You can just get the Illiad from the Gutenberg Project.
      - gmuslera 4y ago
        Every year there are books that end into public domain, either because they were published long enough or they were released in that way recently. Are every one of them in the Gutenberg project? Some of those book even are in their own github project, but it would be nice to have them somewhere, and maybe in a way that could resist censorship o blocking. Or other things than books, like projects or research released in public that some commercial platform centralizes and put a price tag to access them.

wrs 4y ago

Considering the goals of IPFS, I’m surprised it has trouble at this scale. I mean, it’s supposed to be “interplanetary” but this data would fit on two hard drives. The post talks about performance problems with advertising “hundreds” of content IDs. Is it a design problem or just an early implementation problem?

GauntletWizard 4y ago
Design. IPFS is a dead end in design because it's properties are designed with statistical perfection to it's anonymity. This can only ever be achieved statistically, because there's always the attack of "Start Removing nodes at random, and when the content disappears when you remove a node that was your one" This is a provably unsolvable problem, but it's okay: nobody actually has this problem but the worst social outcasts. In some places that's itself a problem: what "The worst social outcasts" is varies from place to place, but running IPFS is enough proof for a death sentence in those places.
- dale_glass 4y ago
  You might be confusing IPFS with something else, like Freenet. As far as I know, there's no such statistical anonymity in it.
  - csande17 4y ago
    Yes, it's fairly easy for attackers to discover the IP address of the people currently hosting a particular piece of content on IPFS: https://docs.ipfs.tech/concepts/privacy-and-encryption/#what-s-public-on-ipfs https://docs.ipfs.tech/concepts/privacy-and-encryption/#what... I think the parent comment is asserting that even if you run IPFS over Tor or something, or even if they try and make IPFS more anonymous in the future, it will always still be vulnerable to the "cut off your Internet and see which node in the network goes down" attack.
ReaLNero 4y ago
Decentralization is expensive e.g. downloading a Google Drive file is immediate while a torrent takes some time initialize connections, or the PoW protocol for cryptocurrencies. Perhaps we could narrow down the purpose of IPFS to storing a small amount of data in a database that is impossible to delete and difficult to censor?

russellbeattie 4y ago

A few months ago I decided on a whim that I should download and de-drm my Audible audiobooks. I have a lot of them from a decade+ of monthly credits being used. I hadn't really considered how much space it would take up so I just set off the download process and did other stuff. 650 audiobooks took up nearly 2TB of space before I stopped it, only noticing when my Mac gave me a warning about not having enough space left on the drive. Whoops. (Audible keeps track of your total listening time... It adds up to over 9 months solid. Crazy, but I can think of worse ways to spend that time.) Anyways, if 6 million ebooks takes up 31TB, I wonder how much space the equivalent amount of audiobooks would use? And how much space a decent library of movies has? I'm kinda thinking that guy on reddit who somehow got a hold of a used Netflix edge server was on to something.

getcrunk 4y ago
the only way i could get that number to work is : 650 books x 15hrs x 3600 seconds x 320kbit/s divided by 8 bits/byte = 1.4tb id say change the audio format and cut the bit-rate roughly in half. but welcome to data hoarding, start with a 8tb external wd dirve!
- russellbeattie 4y ago
  Hmm... well, I do tend to try to maximize the amount of listening I get per credit, so I have lots of 20+ hour books. Stephen Fry's reading of the complete works of Sherlock Holmes is nearly 63 hours [1] and that's just a "single" audiobook (and highly recommended). Lots of Great Courses recordings are similar. Rise and Fall of the Third Reich is 57 hours, etc. Also, there's a ton of metadata like images and PDFs, though I'm not sure how much that takes up. And in fact (just checking again) I'm at only 638 books, so it was probably more like 625 books when I did this. Apparently it adds up. 1. https://www.audible.com/pd/Sherlock-Holmes-Audiobook/B06WLMWF2S https://www.audible.com/pd/Sherlock-Holmes-Audiobook/B06WLMW...
- summm 4y ago
  Audiobooks are usually mono, so 700 kbit/s for uncompressed CD quality. To me parent's numbers sounds like uncompressed. I suspect something went wrong with de-drm/decoding. Should be more like 48 kbit/s for basically transparent mono voice with opus. At this rate, 9 solid months will fit into 142 GB...

yonatron 4y ago

I haven't seen one commenter here concerned with copyright violation or piracy. You're ok with that? Have you never created content of your own, made of your own blood, sweat and tears, and then had it pirated and given away free? Do you condone this? Is this not thievery in your eyes?

RobotToaster 4y ago
>You're ok with that? Yes
getcrunk 4y ago
the problem is that, in general that content made of "blood,sweat and tears" is already being pillaged by the robber barons who take 90%+ of the profit from your work, at least in the fields of movies, tv, music and books. Not saying anything is right or wrong. just pointing out this complicating factor.
- timmytokyo 4y ago
  So by this logic, it's ok to further immiserate the poor author since she was already 90% immiserated by the big bad publisher?
- acdha 4y ago
  There’s a serious citation needed on those numbers but even if they are accurate, theseauthors are not slaves. They voluntarily sign contracts rather than give their work away, self-publish, or stop writing. It seems rather self-serving to argue that you are better equipped to make that decision for them and that means you should get something you want without paying for it.
deleted 4y ago
[deleted]
scotty79 4y ago
Yes. How about you? How hard was your life destroyed by book piracy (or any other kind)?
4y ago

QuadrupleA 4y ago

> It was sad day for the free flow of information, knowledge, and culture. Quick counterpoint to this type of revolutionary rhetoric - writing a book (a good one) takes a lot of time, patience, and hard work. Making it available for free, against the author's wishes, while very easy in the digital age and obviously great for readers, is robbing from the author (if they're still alive). It devalues the craft of writing and makes it even more un-viable as a profession. This may be inevitable, and it has mostly happened with music, journalism, etc., but by putting energy into this kind of project you're only furthering its death along, and perhaps furthering the death of society and culture a little by taking away a lot of the incentive to write anything of value. Pirate away if you want, but thinking this is some kind of virtuous revolutionary act is narcissistic BS.

js8 4y ago
I think you should read https://lessig.org/product/free-culture/ https://lessig.org/product/free-culture/, maybe you will change your mind.
- criddell 4y ago
  Do you see any irony that you are pointing to what is basically an ad for a non-free book?
  - hbbio 4y ago
    On March 25, 2004, Professor Lessig released a PDF version of this book under a Creative Commons license. This HTML version, produced by José Menéndez, is released under the same license. https://www.ibiblio.org/ebooks/Lessig/Free_Culture/Free%20Culture.htm https://www.ibiblio.org/ebooks/Lessig/Free_Culture/Free%20Cu...
- dudul 4y ago
  I'm surprised I need to pay to read this book :)
enriquto 4y ago
Just don't read Kafka then.

js8 4y ago

I wish there was some "library request" mechanism for shadow libraries. I don't mind sharing torrents, especially for older and rare works (which are unlikely to be targeted for copyright), but I can't keep all of them active at the same time, lot of them I have on backup disks. I imagine the client could request for data to become available, then server (person managing a part of shadow library) could make somehow available the files after a certain period, and the client would automatically download the available files.

29athrowaway 4y ago

And this is exactly why I will never use IPFS. I don't want anything to do with anything illegal.

mdp2021 4y ago
Checked https://www.ipfs.tech/ https://www.ipfs.tech/ , it seems like a progressive project - "those things that do something good". Unless you mean that you do not want to have anything to do with hammers and screwdrivers.
- 29athrowaway 4y ago
  Gift cards are a great idea. But who are the #1 users of gift cards right now? Scammers who found in gift cards a good way to conceal their identity and financial info. Today, thousands of elderly people will be scammed by idiots that spend their entire day preying on older people. Elderly people that will lose everything and end up homeless. Fuck the current implementation of gift cards. They cause more problems than they fix. We can't have nice things because there are people out there who are fucking disgusting. Therefore we need accountability built into each technology. Why? because there's a lot of people that fucking suck that's why. Build a simple application that allows people to share plain text and figure out what happens next. Harassment, death threats, organized crime, and all sorts of horrible things. Technocrats and tech enthusiasts are often too idealistic to think about the real world consequences of technology. The person that was having fun implementing convolutional neural networks and decided to publish their research for the benefit of the world would never have imagined that years later the chinese would take that research and build a mass surveillance system that tracked the entire population in real time enslaving billions of people. The people that built Tor would never have imagined the caliber of fucked up shit that now takes place there. And the same with every technology.
  - mdp2021 4y ago
    > gift cards ... accountability built into the technology It depends on what you mean with that «accountability built into the technology». Related case, as you are also speaking of cards: this friend of mine wanted to buy a foreign book (less available in bookstores, where you pay cash). So he purchased an anonymous pre-paid credit card. He found out that during the mess of the past three years, somebody has made it very difficult to keep those cards anonymous: they now request a telephone number to activate them, and in regions where, "coincidentally", telephone numbers are to be associated to legal identities. Now: as you should know, you can bet your life that if our transactions were recorded, we would strictly avoid commerce. (Just like, pretty similarly, we would strictly avoid a non anonymous Internet.) So: your call for accountability should work within the boundaries of what is acceptable. > too idealistic Don't worry, we grow up, and some of us restrain production - exactly in the awareness of what may happen.

dale_glass 4y ago

I'm fairly sure this isn't really safe or really reliable. To my knowledge, IPFS isn't really private, in that both the nodes hosting content can be easily known, and the users requesting content can be monitored. This is bad news for something law enforcement has already taken a serious interest in. IPFS also requires "pinning", which means that unless other people decide to dedicate a few TB to this out of their own initiative, what we have currently is a single machine providing data through an obscure mechanism. If this machine is taken down, the content goes with it. The amount of people that have 31 TB worth of spare storage, care about this particular issue, and are willing to get into legal trouble for it (or at least anger their ISP/host) is probably not terribly large. The work could be split up, but then there needs to be some sort of coordination to somehow divide up hosting the archive among a group of volunteers.

armchairhacker 4y ago
> To my knowledge, IPFS isn't really private, in that both the nodes hosting content can be easily known, and the users requesting content can be monitored. This is bad news for something law enforcement has already taken a serious interest in. You can access the data through a VPN. If necessary, the hosts can also encrypt the filenames and data so that, until law enforcement gets the encryption key, they can't know who accesses what (public key and other necessary info would be communicated through Signal). Rotate the filenames so so when one is discovered, past requests can't be tracked. Maybe there is a way to slightly break the protocol to further hide the requests. > IPFS also requires "pinning", which means that unless other people decide to dedicate a few TB to this out of their own initiative, what we have currently is a single machine providing data through an obscure mechanism. If this machine is taken down, the content goes with it. Do you have to explicitly choose what data to pin? If so then this is an issue. If not, and you just pin random chunks, then if we normalize people using IPFS and distributing legal data this will be solved. If we normalize it enough, there will be too many people hosting and using IPFS for law enforcement to reasonably take down. Or we could just have enough activists that are willing to risk being fined or arrested. --- That being said though, I'm still not convinced on IPFS because it seems like it cannot handle much and is excessively inefficient (case in point: this article). The authors of IPFS should release a new protocol which addresses issues like the article's, hopefully before too much adoption.

tkiolp4 4y ago

I love this but I don’t understand one thing. It’s relatively easy to find “Anna’s Archive” on the internet, and although it doesn’t host copyrighted content itself, it does link to websites that do host copyrighted PDFs, epubs, etc. What’s stopping the FBI (or others) from taking down such linked websites? Isn’t just a matter of time until all of them are taken down and the owners sued?

TheDesolate0 4y ago

Sugimot0 4y ago

What advantage does ipfs hold over tor or i2p? it seems like ipfs is a poor fit for this use case, but I'm not very familiar with these things.

boredhedgehog 4y ago

I would have expected there to be some more effort to curate and cull the collection before rehosting it in full size. Z-Library's automatic deduplication was at best perfunctory; too many works were featured in various formats and editions with no indication of their merits or distinctions. I wouldn't be surprised if the file size could be cut in half without any valuable information lost.

luuuzeta 4y ago

Funnily enough Z-Library made it easy for me to buy physical copies of books I wouldn't have bought otherwise.