5 ms·

> not only uses a hash tree, but it forms a hash tree for every file in the torrent [...] > Files that are identical can also more easily be identified across

by tomxor 6y ago

> not only uses a hash tree, but it forms a hash tree for every file in the torrent [...] > Files that are identical can also more easily be identified across different swarms, since their root hash only depends on the content of the file. Wait, content addressed blobs across swarms... does that mean torrents made by completely different people at different times that happen to contain one or more identical files can benefit from each other's peers? If so this feels like a significant feature that would boost the long term health of a lot of torrents and connect more peers that could be helping each other.

the8472 6y ago
A client can do that optimization locally, but there's no discovery mechanism in the protocolto find additional infohashes that contain some of the same files . But I suppose torrent indexing sites could offer something like that.
Thorentis 6y ago
This right here is the biggest new feature in my opinion, and should have been discussed in more depth in the announcement. I also think they should have pursued this avenue further before releasing v2. Content addressing is one of the big advantages of p2p applications, and IPFS has been pushing it for a long time. Just imagine if this was done all the way down to the piece level. You have two different torrents (say, Linux ISOs) where 20% of the pieces overlap due to similarity (maybe a point upgrade or something and you want both versions). Rather than 100% of both, you only need to download the shared pieces once. Not only that, but say that the latest version has many more seeds/peers, you could download the pieces from that swarm instead, saving the bandwidth of the older torrent's swarm for the remaining 80% you need to download. This could probably be done client side somehow, but it would be good to see actual protocol support for it so that bittorent can move in that direction.
- Ericson2314 6y ago
  Hopefully it would also be good for interopt between the two, as anything with a small enough block size could be picked up by IPLD.
- sbierwagen 6y ago
  >Just imagine if this was done all the way down to the piece level. I don't know about that. With a reasonably large number of users, wouldn't you start seeing hash collisions via birthday paradox? Might be better off keeping it at the file level. (Though this does incentivize malicious users to find hash collisions of particular files they want to defend, and seeding the swarm with garbage files)
  - doomrobo 6y ago
    Hashes are now 256 bits. In order to get a collision with probability 1% you need to produce about 4e37 hashes. For comparison, if you had a 5GHz computer that did a SHA-256 hash every cycle and gave 100 of these computers to every human on Earth, it'd take over 300 million years to produce 4e37 hashes.
    - ShinTakuya 6y ago
      Yep, basically while it's entirely possible (as per Murphy's Law) that there'll be an accidental collision, the likelihood is close to zero. It's a fair risk to be taking.
    - fenomas 6y ago
      > In order to get a collision with probability 1% you need to produce about 4e37 hashes Does "get a collision" here mean finding a particular collision with a known hash, or finding any collision between any two of the generated hashes?
      - doomrobo 6y ago
        The latter. I used the formula from the Wikipedia page on the birthday attack: p ~= 1 - e^{-n^2/2H} where n is the number of items you've picked and H is the size of the universe
        fenomas 6y ago
        I see, thanks!
- sedatk 6y ago
  Only if blocks have the same offset in the same binary and if they align with the block boundaries. Otherwise, different hashes would be generated. I don't expect that to happen a lot.
  - naikrovek 6y ago
    Well if files in an ISO are aligned to some boundary it would help a lot, similar to how filesystems on disk have a sector size, where all files begin at the start of a sector. However, I don't know if this is true in ISO9660 or any of its extensions.
    - em-bee 6y ago
      ISO is a mountable filesystem so this could be the case if it doesn't have any space saving optimizations that create variable block sizes. also need to watch out for compression
  - computerphage 6y ago
    It will still definitely help some use cases: > Identical files will always have the same hash and can more easily be moved from one torrent to another (when creating torrents) without having to re-hash anything.
  - aurelian15 6y ago
    There are ways around this. See "content-aware chunking", e.g. implemented using rolling hashes [1]. This is for example what rsync does. The idea is to make blocks (slightly) variable in size. Block boundaries are determined based on a limited window of preceding bytes. This way a change in one location will only have a limited impact on the following blocks. [1] https://en.wikipedia.org/wiki/Rolling_hash https://en.wikipedia.org/wiki/Rolling_hash
    - octoberfranklin 6y ago
      Rolling hashing is really only useful for finding nonaligned duplicates. There isn't a way to advertise some "rolling hash value" in a way that allows other people with a differently-aligned copy to notice that you and them have some duplicated byte ranges. Rolling hashes only work when one person (or two people engaged in a conversation, like rsync) already has both copies.
terrortrain 6y ago
I haven't looked at it for a while, but wasn't that the big feature of ipfs?
- runeks 6y ago
  Yep. But the difference is that everyone uses BitTorrent and no one uses IPFS.
aaron695 6y ago
> completely different people at different times that happen to contain one or more identical files Yes, this won't be useful for speed. But it will be useful for health and appearance of health. So a torrent that has no seeds because it's missing "RARBG_DO_NOT_MIRROR.exe" can be filled in quickly and have proper seed info. Also large seasons can be filled in. They get corrupted at later episodes because people will prioritise the first episodes then the seeds can jump. So anyone filling in the later episodes through single episodes torrents might repair the season torrent. It'll allow over lap between torrents only differentiated be a missing RARBG.txt if they are both fragmented. There's always been talk of allowing a torrent to have files added to it. It's be interesting to see if this might be useful client side to do this in a Claytons way. Personally I don't think it's useful for Linux isos. Not sure on this. Do Linux isos get unhealthy?
- em-bee 6y ago
  patch releases, especially security patches would only have a small number of changes that might allow me to take a full new release but only update the changes to the release i already have. but it would take some handcrafting to make sure the new files are at the end of the ISO so everything else stays in place. technically this should be possible since ISO should allow for adding files given that it was designed for write-once media.
yc12340 6y ago
> Wait, content addressed blobs across swarms... does that mean torrents made by completely different people at different times that happen to contain one or more identical files can benefit from each other's peers? Yes. It also makes your files more discoverable for copyright trolls. They will be able to fully automate the process of sending takedown notices for each copy of zlib's README.txt, that was "stolen" from their software by evil pirates.
knodi123 6y ago
> this feels like a significant feature that would boost the long term health of a lot of torrents Except for so many that are a single archive file. It seems logical that you'd want to compress what you're sending as much as possible before uploading... but in today's bandwidth environment, it actually makes sense to leave certain things uncompressed so you can benefit from swarm overlaps?!? Wild!
- abluecloud 6y ago
  do two identical compressions produce the same output? if so, wouldn't the same thing happened just against the compressed files? It's not uncommon to see multiple .torrent files with the same content FYI.
  - AnIdiotOnTheNet 6y ago
    Normally no, compression formats like zip are not guaranteed to produce the same output for the same input. That's why things like torrentzip and torrent7zip were created, to ensure that the same input did create the same output.

by tomxor 6y ago

the8472 6y ago

A client can do that optimization locally, but there's no discovery mechanism in the protocolto find additional infohashes that contain some of the same files . But I suppose torrent indexing sites could offer something like that.

Thorentis 6y ago

This right here is the biggest new feature in my opinion, and should have been discussed in more depth in the announcement. I also think they should have pursued this avenue further before releasing v2. Content addressing is one of the big advantages of p2p applications, and IPFS has been pushing it for a long time. Just imagine if this was done all the way down to the piece level. You have two different torrents (say, Linux ISOs) where 20% of the pieces overlap due to similarity (maybe a point upgrade or something and you want both versions). Rather than 100% of both, you only need to download the shared pieces once. Not only that, but say that the latest version has many more seeds/peers, you could download the pieces from that swarm instead, saving the bandwidth of the older torrent's swarm for the remaining 80% you need to download. This could probably be done client side somehow, but it would be good to see actual protocol support for it so that bittorent can move in that direction.

Ericson2314 6y ago
Hopefully it would also be good for interopt between the two, as anything with a small enough block size could be picked up by IPLD.
sbierwagen 6y ago
>Just imagine if this was done all the way down to the piece level. I don't know about that. With a reasonably large number of users, wouldn't you start seeing hash collisions via birthday paradox? Might be better off keeping it at the file level. (Though this does incentivize malicious users to find hash collisions of particular files they want to defend, and seeding the swarm with garbage files)
- doomrobo 6y ago
  Hashes are now 256 bits. In order to get a collision with probability 1% you need to produce about 4e37 hashes. For comparison, if you had a 5GHz computer that did a SHA-256 hash every cycle and gave 100 of these computers to every human on Earth, it'd take over 300 million years to produce 4e37 hashes.
  - ShinTakuya 6y ago
    Yep, basically while it's entirely possible (as per Murphy's Law) that there'll be an accidental collision, the likelihood is close to zero. It's a fair risk to be taking.
  - fenomas 6y ago
    > In order to get a collision with probability 1% you need to produce about 4e37 hashes Does "get a collision" here mean finding a particular collision with a known hash, or finding any collision between any two of the generated hashes?
    - doomrobo 6y ago
      The latter. I used the formula from the Wikipedia page on the birthday attack: p ~= 1 - e^{-n^2/2H} where n is the number of items you've picked and H is the size of the universe
      - fenomas 6y ago
        I see, thanks!
sedatk 6y ago
Only if blocks have the same offset in the same binary and if they align with the block boundaries. Otherwise, different hashes would be generated. I don't expect that to happen a lot.
- naikrovek 6y ago
  Well if files in an ISO are aligned to some boundary it would help a lot, similar to how filesystems on disk have a sector size, where all files begin at the start of a sector. However, I don't know if this is true in ISO9660 or any of its extensions.
  - em-bee 6y ago
    ISO is a mountable filesystem so this could be the case if it doesn't have any space saving optimizations that create variable block sizes. also need to watch out for compression
- computerphage 6y ago
  It will still definitely help some use cases: > Identical files will always have the same hash and can more easily be moved from one torrent to another (when creating torrents) without having to re-hash anything.
- aurelian15 6y ago
  There are ways around this. See "content-aware chunking", e.g. implemented using rolling hashes [1]. This is for example what rsync does. The idea is to make blocks (slightly) variable in size. Block boundaries are determined based on a limited window of preceding bytes. This way a change in one location will only have a limited impact on the following blocks. [1] https://en.wikipedia.org/wiki/Rolling_hash https://en.wikipedia.org/wiki/Rolling_hash
  - octoberfranklin 6y ago
    Rolling hashing is really only useful for finding nonaligned duplicates. There isn't a way to advertise some "rolling hash value" in a way that allows other people with a differently-aligned copy to notice that you and them have some duplicated byte ranges. Rolling hashes only work when one person (or two people engaged in a conversation, like rsync) already has both copies.

terrortrain 6y ago

I haven't looked at it for a while, but wasn't that the big feature of ipfs?

runeks 6y ago
Yep. But the difference is that everyone uses BitTorrent and no one uses IPFS.

aaron695 6y ago

> completely different people at different times that happen to contain one or more identical files Yes, this won't be useful for speed. But it will be useful for health and appearance of health. So a torrent that has no seeds because it's missing "RARBG_DO_NOT_MIRROR.exe" can be filled in quickly and have proper seed info. Also large seasons can be filled in. They get corrupted at later episodes because people will prioritise the first episodes then the seeds can jump. So anyone filling in the later episodes through single episodes torrents might repair the season torrent. It'll allow over lap between torrents only differentiated be a missing RARBG.txt if they are both fragmented. There's always been talk of allowing a torrent to have files added to it. It's be interesting to see if this might be useful client side to do this in a Claytons way. Personally I don't think it's useful for Linux isos. Not sure on this. Do Linux isos get unhealthy?

em-bee 6y ago
patch releases, especially security patches would only have a small number of changes that might allow me to take a full new release but only update the changes to the release i already have. but it would take some handcrafting to make sure the new files are at the end of the ISO so everything else stays in place. technically this should be possible since ISO should allow for adding files given that it was designed for write-once media.

yc12340 6y ago

> Wait, content addressed blobs across swarms... does that mean torrents made by completely different people at different times that happen to contain one or more identical files can benefit from each other's peers? Yes. It also makes your files more discoverable for copyright trolls. They will be able to fully automate the process of sending takedown notices for each copy of zlib's README.txt, that was "stolen" from their software by evil pirates.

knodi123 6y ago

> this feels like a significant feature that would boost the long term health of a lot of torrents Except for so many that are a single archive file. It seems logical that you'd want to compress what you're sending as much as possible before uploading... but in today's bandwidth environment, it actually makes sense to leave certain things uncompressed so you can benefit from swarm overlaps?!? Wild!

abluecloud 6y ago
do two identical compressions produce the same output? if so, wouldn't the same thing happened just against the compressed files? It's not uncommon to see multiple .torrent files with the same content FYI.
- AnIdiotOnTheNet 6y ago
  Normally no, compression formats like zip are not guaranteed to produce the same output for the same input. That's why things like torrentzip and torrent7zip were created, to ensure that the same input did create the same output.