17 ms·

This update to the spec is a modest change that's largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-by

This update to the spec is a modest change that's largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-byte length of a SHA1 checksum. They've decided to move forward with SHA256 truncated to 20 bytes to avoid incompatibilities with existing infrastructure such as the Mainline DHT. Beyond the hashing algorithm, some important additions that were previously proposals without widespread use (e.g. merkle tree for hashing pieces) are becoming required. The focus has mostly been on optimizing latency for the P2P protocol and making sane improvements to the file spec. I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Ideally, BitTorrent would be broken down into separate specifications that could be used together or in separate systems: one for the file format and piece representation for sharing files, one for the P2P protocol, and one for discovery (trackers, DHTs). I want to believe that there would be far more interesting P2P projects if you could just lift robust primitives from BitTorrent.

the8472 9y ago
The DHT BEPs specify a network that is only barely related to the bittorrent core protocol, they can already be used independently, and some people do. > I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Yes, we did not pay much attention to trackers, but BEP52 basically seized the opportunity to do some incompatible changes we always wanted to do anyway (quite a few accumulated over the years), and there were no such open issues with the http tracker protocol.
- jzelinskie 9y ago
  >...and there were no such open issues with the http tracker protocol This is because the HTTP protocol is so much overhead that most trackers don't even really run it anymore. I think UDP being promoted to the spec would've been a step in the right direction. Modern trackers have a bunch of tricks like BEP34[0] to avoid getting pounded that would be great if every client conformed to. I hope I'm not coming off as aggressive. I really appreciate this work and I'm really glad to see a spec revision. It's just as you said, there's been many years and many good improvements that I'd like to see made while there's still a change to break things. [0]: http://www.bittorrent.org/beps/bep_0034.html http://www.bittorrent.org/beps/bep_0034.html
  - the8472 9y ago
    I still don't see why this would need to piggyback on a breaking change of the metadata file format. "Please use UDP by default" is fairly orthogonal to the metadata format and could be added to the spec at any point if we want to. HTTP trackers are considered fine for medium-scale torrent deployments. UDP trackers were originally introduced to cope with the traffic caused by running an open tracker that manages 100k+ infohashes for the whole world. Also, both BEP3 and 52 already forward-reference the tracker extensions (compact and UDP), so someone who writes a new bittorrent implementation should already be aware of them. Maybe we could make it more clear that some BEPs are almost-mandatory.
- predakanga 9y ago
  How do you intend to handle incompatibilities in the tracker protocol that are introduced by BEP52? In particular, I note that there's nothing in there regarding which infohash should be used in the tracker updates. Should traffic with v1/v2 clients be reported separately, or should it be consolidated under the v2 infohash?
  - the8472 9y ago
    hybrid torrents essentially consist of two swarms, so you announce twice.
- aninhumer 9y ago
  >The DHT BEPs specify a network that is only barely related to the bittorrent core protocol Yeah, if I remember correctly bittorrent DHT ultimately just maps 20 byte hashes to peer-lists (IP + port pairs). It's obviously designed to be convenient for bittorrent swarm discovery, but nothing about it limits it to bittorrent usage. Indeed, I'm surprised it's not more widely exploited for p2p bootstrapping.
- pdimitar 9y ago
  More generic question, apologies if it feels inserted without relevance: Did you guys talk with the IPFS team? Do both of you have a desire to start bringing both families of protocols and technologies closer together? I feel in this age we must make de-fragmentation of efforts our topmost priority.
userbinator 9y ago
SHA256 truncated to 20 bytes What are the security implications of doing this? It seems it wouldn't increase the strength beyond the original 160 bits, no? Was there anything preventing redesigning the protocol to use full 32-byte SHA256 hashes throughout?
- scrollaway 9y ago
  The problem with SHA1 being broken is unrelated to the amount of bits in the hash, so the point of switching to sha256 isn't to "increase the strength beyond the original 160 bits" but to avoid Shattered and future potential attacks.
- sha2throwaway 9y ago
  Actually you do get some added security, because it prevents the length extension attacks sha-2 and related hash functions have thanks to the Merkle-Damgard construction [0]. Specifically, by truncating the hash, the output no longer contains enough state to perform the attack. [0] https://en.wikipedia.org/wiki/Length_extension_attack https://en.wikipedia.org/wiki/Length_extension_attack
  - loup-vaillant 9y ago
    There are better ways to prevent length extension attacks, such as choosing Blake2b. With the current scheme, only 12 bytes are missing, so Length extension attacks only get 96 bits of security… Replacing SHA-1 with SHA-2, what are they thinking? Blake2 is faster and more secure than either.
    - derefr 9y ago
      They're probably thinking something about which hashes will get hardware-accelerated instructions provided on x86 and ARM.
      - loup-vaillant 9y ago
        Yeah, I keep forgetting about Hardware acceleration…
RJIb8RBYxzAMX9u 9y ago
> They've decided to move forward with SHA256 truncated to 20 bytes [...] I wonder why not SHA512? It's actually faster to compute than SHA512 on 64-bit architectures.
- silotis 9y ago
  SHA256 was chosen over SHA512 because the two most popular 64-bit ISAs (x86-64 and ARMv8) both define instruction extensions to provide hardware acceleration of SHA256. Hardware support makes SHA256 much faster than SHA512 in software, even on 64-bit processors.
  - RJIb8RBYxzAMX9u 9y ago
    Huh, I didn't know that's the case for x86. My quick search suggests that support is still quite limited though, and none of my HW (1~4 yrs old) support it. I suppose by the time the spec is finalized and widely adopted HW-accelerated SHA256 would be, too (and I should switch my zfs checksum back from SHA512/256 to SHA256).
    - AbacusAvenger 9y ago
      I haven't found any Intel hardware with the SHA extensions actually implemented. I was expecting it in Skylake but apparently that's not the case... Not sure when we can expect to see them.
      - silotis 9y ago
        Right now the most common x86 CPU which implements the SHA instructions is AMD's Ryzen. The only available Intel CPU which supports them is Goldmont, the latest generation of Atom CPU. It seems likely that Cannon Lake and possibly Coffee Lake will also implement them, but I haven't seen any rumors pointing one way or the other.
      - mtgx 9y ago
        Intel has recently implemented SHA1 in hardware. So give it another decade or so.
mtgx 9y ago
The word encryption doesn't even seem to be mentioned there. At the very least it would help against traffic shaping, which you know is coming once net neutrality rules are dead.
- the8472 9y ago
  Bittorrent already had an obfuscation protocol wrapper for years. It was effective for a while but the companies that implement traffic shaping equipment stepped up their game and probably rely on traffic flow matching now. It is an arms race that is not won by updating a slowly evolving core protocol.
boyce 9y ago
I'd love to see a p2p wikipedia and a p2p twitter alternative Moving bittorrent away from it's present image could be achieved by making p2p useful beyond bluray rips
- fps_doug 9y ago
  I liked how the old Opera included a bittorrent client. It was the only browser that did, and felt like it was actually the way it was meant to be used. But nobody got it. You had warez kiddies left and right complaining how it sucked compared to Azureus and later µTorrent which had bazillions of features to tweak and max out their connection, or saying it was useless because their favorite ALT did not whitelist Opera. But I just used it a lot when running bigger downloads like install discs for Linux distros, OpenOffice etc., and it made a difference when there was some major release and half of the plain old http mirrors were painfully slow or down entirely. Admittedly, that situation got a lot better compared to 10 years ago, but still I'm delighted by how natural it felt to use, since it seamlessly integrated with the browser's download manager. And you didn't have this "uh, I need to start an external program for this" kind of reluctant thought when you saw a website offered download via torrent. Today I just wonder if BT would have evolved differently if all browsers would have included a client.
  - cesarb 9y ago
    Also, the original Bittorrent client looked like an Internet Explorer file download dialog. The download experience was nearly identical: you clicked on a download link, and a download dialog with a progress bar opened up. Behind the scenes, the torrent file MIME type was registered to the Bittorrent client, so it was downloading the torrent file and launching the Bittorrent client, while a normal download showed the download dialog directly, but the user didn't have to be aware of these details. So yes, it's the way it was originally meant to be used.
  - boyce 9y ago
    I think there's still potential to steer BT in a more exciting direction if there was the will to from those involved. More quality software projects using it to download by default. Bring the trust back, shake off the reputation for just being trojan-infested cracks. There's plenty of things besides piracy people could be doing with torrents and related tech and it seems like such a waste of an idea. A Linux package manager, an open-source Acestream alternative, collaborative work on large scales

the8472 9y ago

The DHT BEPs specify a network that is only barely related to the bittorrent core protocol, they can already be used independently, and some people do. > I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Yes, we did not pay much attention to trackers, but BEP52 basically seized the opportunity to do some incompatible changes we always wanted to do anyway (quite a few accumulated over the years), and there were no such open issues with the http tracker protocol.

jzelinskie 9y ago
>...and there were no such open issues with the http tracker protocol This is because the HTTP protocol is so much overhead that most trackers don't even really run it anymore. I think UDP being promoted to the spec would've been a step in the right direction. Modern trackers have a bunch of tricks like BEP34[0] to avoid getting pounded that would be great if every client conformed to. I hope I'm not coming off as aggressive. I really appreciate this work and I'm really glad to see a spec revision. It's just as you said, there's been many years and many good improvements that I'd like to see made while there's still a change to break things. [0]: http://www.bittorrent.org/beps/bep_0034.html http://www.bittorrent.org/beps/bep_0034.html
- the8472 9y ago
  I still don't see why this would need to piggyback on a breaking change of the metadata file format. "Please use UDP by default" is fairly orthogonal to the metadata format and could be added to the spec at any point if we want to. HTTP trackers are considered fine for medium-scale torrent deployments. UDP trackers were originally introduced to cope with the traffic caused by running an open tracker that manages 100k+ infohashes for the whole world. Also, both BEP3 and 52 already forward-reference the tracker extensions (compact and UDP), so someone who writes a new bittorrent implementation should already be aware of them. Maybe we could make it more clear that some BEPs are almost-mandatory.
predakanga 9y ago
How do you intend to handle incompatibilities in the tracker protocol that are introduced by BEP52? In particular, I note that there's nothing in there regarding which infohash should be used in the tracker updates. Should traffic with v1/v2 clients be reported separately, or should it be consolidated under the v2 infohash?
- the8472 9y ago
  hybrid torrents essentially consist of two swarms, so you announce twice.
aninhumer 9y ago
>The DHT BEPs specify a network that is only barely related to the bittorrent core protocol Yeah, if I remember correctly bittorrent DHT ultimately just maps 20 byte hashes to peer-lists (IP + port pairs). It's obviously designed to be convenient for bittorrent swarm discovery, but nothing about it limits it to bittorrent usage. Indeed, I'm surprised it's not more widely exploited for p2p bootstrapping.
pdimitar 9y ago
More generic question, apologies if it feels inserted without relevance: Did you guys talk with the IPFS team? Do both of you have a desire to start bringing both families of protocols and technologies closer together? I feel in this age we must make de-fragmentation of efforts our topmost priority.

userbinator 9y ago

SHA256 truncated to 20 bytes What are the security implications of doing this? It seems it wouldn't increase the strength beyond the original 160 bits, no? Was there anything preventing redesigning the protocol to use full 32-byte SHA256 hashes throughout?

scrollaway 9y ago
The problem with SHA1 being broken is unrelated to the amount of bits in the hash, so the point of switching to sha256 isn't to "increase the strength beyond the original 160 bits" but to avoid Shattered and future potential attacks.
sha2throwaway 9y ago
Actually you do get some added security, because it prevents the length extension attacks sha-2 and related hash functions have thanks to the Merkle-Damgard construction [0]. Specifically, by truncating the hash, the output no longer contains enough state to perform the attack. [0] https://en.wikipedia.org/wiki/Length_extension_attack https://en.wikipedia.org/wiki/Length_extension_attack
- loup-vaillant 9y ago
  There are better ways to prevent length extension attacks, such as choosing Blake2b. With the current scheme, only 12 bytes are missing, so Length extension attacks only get 96 bits of security… Replacing SHA-1 with SHA-2, what are they thinking? Blake2 is faster and more secure than either.
  - derefr 9y ago
    They're probably thinking something about which hashes will get hardware-accelerated instructions provided on x86 and ARM.
    - loup-vaillant 9y ago
      Yeah, I keep forgetting about Hardware acceleration…

RJIb8RBYxzAMX9u 9y ago

> They've decided to move forward with SHA256 truncated to 20 bytes [...] I wonder why not SHA512? It's actually faster to compute than SHA512 on 64-bit architectures.

silotis 9y ago
SHA256 was chosen over SHA512 because the two most popular 64-bit ISAs (x86-64 and ARMv8) both define instruction extensions to provide hardware acceleration of SHA256. Hardware support makes SHA256 much faster than SHA512 in software, even on 64-bit processors.
- RJIb8RBYxzAMX9u 9y ago
  Huh, I didn't know that's the case for x86. My quick search suggests that support is still quite limited though, and none of my HW (1~4 yrs old) support it. I suppose by the time the spec is finalized and widely adopted HW-accelerated SHA256 would be, too (and I should switch my zfs checksum back from SHA512/256 to SHA256).
  - AbacusAvenger 9y ago
    I haven't found any Intel hardware with the SHA extensions actually implemented. I was expecting it in Skylake but apparently that's not the case... Not sure when we can expect to see them.
    - silotis 9y ago
      Right now the most common x86 CPU which implements the SHA instructions is AMD's Ryzen. The only available Intel CPU which supports them is Goldmont, the latest generation of Atom CPU. It seems likely that Cannon Lake and possibly Coffee Lake will also implement them, but I haven't seen any rumors pointing one way or the other.
    - mtgx 9y ago
      Intel has recently implemented SHA1 in hardware. So give it another decade or so.

mtgx 9y ago

The word encryption doesn't even seem to be mentioned there. At the very least it would help against traffic shaping, which you know is coming once net neutrality rules are dead.

the8472 9y ago
Bittorrent already had an obfuscation protocol wrapper for years. It was effective for a while but the companies that implement traffic shaping equipment stepped up their game and probably rely on traffic flow matching now. It is an arms race that is not won by updating a slowly evolving core protocol.

boyce 9y ago

I'd love to see a p2p wikipedia and a p2p twitter alternative Moving bittorrent away from it's present image could be achieved by making p2p useful beyond bluray rips

fps_doug 9y ago
I liked how the old Opera included a bittorrent client. It was the only browser that did, and felt like it was actually the way it was meant to be used. But nobody got it. You had warez kiddies left and right complaining how it sucked compared to Azureus and later µTorrent which had bazillions of features to tweak and max out their connection, or saying it was useless because their favorite ALT did not whitelist Opera. But I just used it a lot when running bigger downloads like install discs for Linux distros, OpenOffice etc., and it made a difference when there was some major release and half of the plain old http mirrors were painfully slow or down entirely. Admittedly, that situation got a lot better compared to 10 years ago, but still I'm delighted by how natural it felt to use, since it seamlessly integrated with the browser's download manager. And you didn't have this "uh, I need to start an external program for this" kind of reluctant thought when you saw a website offered download via torrent. Today I just wonder if BT would have evolved differently if all browsers would have included a client.
- cesarb 9y ago
  Also, the original Bittorrent client looked like an Internet Explorer file download dialog. The download experience was nearly identical: you clicked on a download link, and a download dialog with a progress bar opened up. Behind the scenes, the torrent file MIME type was registered to the Bittorrent client, so it was downloading the torrent file and launching the Bittorrent client, while a normal download showed the download dialog directly, but the user didn't have to be aware of these details. So yes, it's the way it was originally meant to be used.
- boyce 9y ago
  I think there's still potential to steer BT in a more exciting direction if there was the will to from those involved. More quality software projects using it to download by default. Bring the trust back, shake off the reputation for just being trojan-infested cracks. There's plenty of things besides piracy people could be doing with torrents and related tech and it seems like such a waste of an idea. A Linux package manager, an open-source Acestream alternative, collaborative work on large scales