9 ms·

I'm much more familiar with Bittorrent than IPFS, but as I understand it IPFS treats each object as its own swarm. This is quite the opposite of having a single

I'm much more familiar with Bittorrent than IPFS, but as I understand it IPFS treats each object as its own swarm. This is quite the opposite of having a single global swarm. In Bittorrent terms it's as if each piece and the torrent itself are all separate swarms. What makes IFPS more global is that they have defined a set of standards for linking objects together to form potentially complex structures. The holy grail is to scale this up to create a whole new World Wide Web based on linked IFPS objects rather than linking to resources on specific hosts.

setr 8y ago
Its still not clear to me if its true, but my impression from reading the whitepaper a couple years back was that you could fetch hashblocks for your target file, from any arbitrary node that happens to be serving it, regardless of what the original target was. Eg a block composed purely of white pixels can be equally shared across thousand of gifs, independent of the original file. Which if true would make it more of a global swarm Oddly, it would also imply you could potentially generate entirely new files out of the network; in an infinitely large network, you wouldnt even have to specifically upload a new file — the hashtree would be enough, and the network would find the pieces to compose it Never got anyone to confirm/deny my understanding of it, and its not clear to me why it wouldnt be the case, if it weren’t.
- krrrh 8y ago
  Given that there would never be an infinitely large network, how likely is it that this sort of de-duplication would offer benefits? We use hashes to ensure data integrity because if a single bit is changed in the data the hash will reliably change. GIF compression already reduces "256 white pixels in a row" to a few bytes (other compression algorithms aren't so straightforward). Even with all the white gifs people are sharing online, you'd need vanishingly small block sizes for this strategy to achieve anything measurable, approaching the point where the hash doesn't just represent the block, but is the block.
  - setr 8y ago
    It's more of it's not clear to me why you wouldn't implement it in this fashion, my impression from the paper was that it was implemented in this fashion, and as a side-effect of that implementation, this kind of de-duplication would be possible somewhat by accident. More specifically, its not clear to me what benefit exists to associating a block with its originating file; my understanding of bittorrent requires you to receive/send out the total list of blocks anyways (the .torrent file), as you're requesting the missing blocks (and at start, thats all blocks), and others are responding with blocks the have to offer (which may not be all blocks, since you can request from partially-finished peers aka leechers). In IPFS, you don't have an associated server (a tracker) to tell you who your peers are for a given file, so first you'll ask around for the list of blocks required for that file, and then you'll ask anyone who responded for your missing blocks, same as bittorrent. But you'll have to keep asking for new peers, since you aren't being updated by the server, and sending out the list of missing blocks all the same. It seems to me that there's not much benefit to requesting specifically the file, and then the blocks, rather than just requesting the blocks directly, when doing that updated request. And if you're asking for the blocks directly... then that kind of de-duplication would be a natural outcome. Also a spot where it might actually be impactful today -- Memes. And if its truly an interplanetary mindset, there might come a point long into the future where such a feature would be an actual feature, rather than an accident. But regardless, I only mean it as an implication of how I believe IPFS is implemented, not an intended benefit/feature of it. I specifically mean it as an easy way to point out where my understanding failed, if I'm wrong; or an easy way to verify my understanding is correct ;)
- perfmode 8y ago
  I wrote the original data exchange (bitswap) code. Your first paragraph is true.
- perfmode 8y ago
  > Oddly, it would also imply you could potentially generate entirely new files out of the network; in an infinitely large network, you wouldnt even have to specifically upload a new file — the hashtree would be enough, and the network would find the pieces to compose it It's vanishingly unlikely that this would yield useful results in practice. In a moderately-sized network, however, if only a few contiguous bytes in a GB file change, then only a single block would need to be uploaded. (if the original file is sufficiently available already)

setr 8y ago

Its still not clear to me if its true, but my impression from reading the whitepaper a couple years back was that you could fetch hashblocks for your target file, from any arbitrary node that happens to be serving it, regardless of what the original target was. Eg a block composed purely of white pixels can be equally shared across thousand of gifs, independent of the original file. Which if true would make it more of a global swarm Oddly, it would also imply you could potentially generate entirely new files out of the network; in an infinitely large network, you wouldnt even have to specifically upload a new file — the hashtree would be enough, and the network would find the pieces to compose it Never got anyone to confirm/deny my understanding of it, and its not clear to me why it wouldnt be the case, if it weren’t.

krrrh 8y ago
Given that there would never be an infinitely large network, how likely is it that this sort of de-duplication would offer benefits? We use hashes to ensure data integrity because if a single bit is changed in the data the hash will reliably change. GIF compression already reduces "256 white pixels in a row" to a few bytes (other compression algorithms aren't so straightforward). Even with all the white gifs people are sharing online, you'd need vanishingly small block sizes for this strategy to achieve anything measurable, approaching the point where the hash doesn't just represent the block, but is the block.
- setr 8y ago
  It's more of it's not clear to me why you wouldn't implement it in this fashion, my impression from the paper was that it was implemented in this fashion, and as a side-effect of that implementation, this kind of de-duplication would be possible somewhat by accident. More specifically, its not clear to me what benefit exists to associating a block with its originating file; my understanding of bittorrent requires you to receive/send out the total list of blocks anyways (the .torrent file), as you're requesting the missing blocks (and at start, thats all blocks), and others are responding with blocks the have to offer (which may not be all blocks, since you can request from partially-finished peers aka leechers). In IPFS, you don't have an associated server (a tracker) to tell you who your peers are for a given file, so first you'll ask around for the list of blocks required for that file, and then you'll ask anyone who responded for your missing blocks, same as bittorrent. But you'll have to keep asking for new peers, since you aren't being updated by the server, and sending out the list of missing blocks all the same. It seems to me that there's not much benefit to requesting specifically the file, and then the blocks, rather than just requesting the blocks directly, when doing that updated request. And if you're asking for the blocks directly... then that kind of de-duplication would be a natural outcome. Also a spot where it might actually be impactful today -- Memes. And if its truly an interplanetary mindset, there might come a point long into the future where such a feature would be an actual feature, rather than an accident. But regardless, I only mean it as an implication of how I believe IPFS is implemented, not an intended benefit/feature of it. I specifically mean it as an easy way to point out where my understanding failed, if I'm wrong; or an easy way to verify my understanding is correct ;)
perfmode 8y ago
I wrote the original data exchange (bitswap) code. Your first paragraph is true.
perfmode 8y ago
> Oddly, it would also imply you could potentially generate entirely new files out of the network; in an infinitely large network, you wouldnt even have to specifically upload a new file — the hashtree would be enough, and the network would find the pieces to compose it It's vanishingly unlikely that this would yield useful results in practice. In a moderately-sized network, however, if only a few contiguous bytes in a GB file change, then only a single block would need to be uploaded. (if the original file is sufficiently available already)