11 ms·

The BitTorrent Protocol Specification v2

319 points by Nekit1234007 9y ago

lowglow 9y ago
Can someone diff the spec from the previous version? What's the changelog? :)
- mouldysammich 9y ago
  The main differences I can see is a change from SHA1 -> SHA2 and also seems to have added official spec for webtorrent.
  - Luminarys 9y ago
    It also appears to be using a merkle hash tree for piece hashing now along with a few new peer wire messages to support that.
    - phire 9y ago
      It also switches to a merkle hash per-file and specifies that each file is aligned to the start of a piece, and the size of the last piece of each file matches the amount of remaining data. This means in large multi-file torrents you don't have to download (and store) the two extra 1-4mb pieces at the start/end of each file anymore.
  - cjbprime 9y ago
    > official spec for webtorrent Huh, where do you see that? Not seeing any ctrl-f hits for webtorrent or webrtc.
    - mouldysammich 9y ago
      Whoops, i skimmed and they mentioned the end user web browser and I assumed they had. I guess this is why they say when you assume you make an ass out of you and me.
- kpcyrd 9y ago
  It would be interesting to see how the new version compares to ipfs.
  - supergreg 9y ago
    The tree structure seems very similar. It would be nice if torrent clients could interact with ipfs or gain ipfs capabilities. Think torrents that update themselves when the files change (thanks to ipns).
    - sktrdie 9y ago
      There's already such BEP: http://bittorrent.org/beps/bep_0046.html http://bittorrent.org/beps/bep_0046.html - "Updating Torrents Via DHT Mutable Items"
jzelinskie 9y ago
This update to the spec is a modest change that's largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-byte length of a SHA1 checksum. They've decided to move forward with SHA256 truncated to 20 bytes to avoid incompatibilities with existing infrastructure such as the Mainline DHT. Beyond the hashing algorithm, some important additions that were previously proposals without widespread use (e.g. merkle tree for hashing pieces) are becoming required. The focus has mostly been on optimizing latency for the P2P protocol and making sane improvements to the file spec. I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Ideally, BitTorrent would be broken down into separate specifications that could be used together or in separate systems: one for the file format and piece representation for sharing files, one for the P2P protocol, and one for discovery (trackers, DHTs). I want to believe that there would be far more interesting P2P projects if you could just lift robust primitives from BitTorrent.
- the8472 9y ago
  The DHT BEPs specify a network that is only barely related to the bittorrent core protocol, they can already be used independently, and some people do. > I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Yes, we did not pay much attention to trackers, but BEP52 basically seized the opportunity to do some incompatible changes we always wanted to do anyway (quite a few accumulated over the years), and there were no such open issues with the http tracker protocol.
  - jzelinskie 9y ago
    >...and there were no such open issues with the http tracker protocol This is because the HTTP protocol is so much overhead that most trackers don't even really run it anymore. I think UDP being promoted to the spec would've been a step in the right direction. Modern trackers have a bunch of tricks like BEP34[0] to avoid getting pounded that would be great if every client conformed to. I hope I'm not coming off as aggressive. I really appreciate this work and I'm really glad to see a spec revision. It's just as you said, there's been many years and many good improvements that I'd like to see made while there's still a change to break things. [0]: http://www.bittorrent.org/beps/bep_0034.html http://www.bittorrent.org/beps/bep_0034.html
0x0 9y ago
What's the stuff about "proof layers", is that new in this v2? The paper briefly talks about proof layer requests. Is this something merkle-tree related? What is the purpose? Is it to prevent clients from lying about having pieces they do not have by requesting a verifiable random hash chunk?
- the8472 9y ago
  It's part of switching to merkle trees instead of flat piece lists. A merkle tree can only be verified if you either have a whole layer or send ancestor-siblings (uncle, great-uncle, etc.) along with a partial layer. Merkle trees allow torrents to start faster from magnet links since only the tree roots need to be front-loaded while the tree can be fetched incrementally.
shmerl 9y ago
Do all Bittorrent clients support it already?
- silotis 9y ago
  Currently no bittorrent clients support it. This is still just a draft. I've only just started working on an implementation for libtorrent, it will be quite a while before it is production ready.
richdougherty 9y ago
Rationale for hash function change: https://github.com/bittorrent/bittorrent.org/issues/58 https://github.com/bittorrent/bittorrent.org/issues/58 Discussion of other changes: https://github.com/bittorrent/bittorrent.org/pull/59 https://github.com/bittorrent/bittorrent.org/pull/59
Scaevolus 9y ago
1) Chunks don't span files. Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwards compatible-- you can make a .torrent file with both old and new pieces, and a swarm can speak either. This requires padding files from BEP47, which most clients probably don't support. Per-file metadata increases pretty significantly, from ~19B (just length) to ~68B (length + hash).
- phire 9y ago
  Per-file metadata increases significantly, but it gets rid of the per piece data (which in bittorrent v1 is 20 bytes of sha1 hash per piece and made up the bulk of the .torrent file). The .torrent file only stores the merkle tree's root hash for each file, and the torrent client will query it's peers to get the rest of the merkle tree (verifiable against the root hash). The leafs of the merkle tree are the hash of each 16kb block. Interesting consequences of this: Piece size isn't baked into the file anymore (and I've seen torrents with 16mb blocks), the client can dynamically chose it's verification piece size by requesting only so many layers of the merkle tree. Or it could skip requesting the tree and verify the whole file at once. Merkle tree roots will be globally unique. You can scan torrent files for duplicated files and download common files from multiple swarms.
  - computerphage 9y ago
    > Or it could skip requesting the tree and verify the whole file at once. To clarify, this works by the client deterministically reconstructing the tree once they have the whole file, then checking the root's hash, correct?
    - phire 9y ago
      Yeah, it's a deterministic tree of hashes. Each leaf is the hash of a 16KB chunk. On the next layer up you have a series of nodes which are the hashes of the two leafs below it hashed together. You add enough layers until you get a single root hash at the root of the tree.
smegel 9y ago
Pity we will never see a genuine version of uTorrent that will support it. That was a real loss.
- vanderZwan 9y ago
  We have plenty of good open source alternatives now. qTorrent works fine
  - smegel 9y ago
    Thanks I haven't hear of that. And it is written in C++ which is nice. Is it considered the spiritual successor to the original uTorrent?
    - nyolfen 9y ago
      it is: >The qBittorrent project aims to provide an open-source software alternative to µTorrent. though in my experience it is more of a memory hog and buggier than utorrent. but that doesn't stop me from using it
      - lucasverra 9y ago
        I've switched to transmission client and never looked back (4 yrs ago)
      - phire 9y ago
        It's nowhere near as perfect as utorrent once was. But it's pretty good and I have plenty of extra memory (it's currently using 305mb while seeding 40+ torrents, dropping down to 65mb if I pause them all.)
    - vopi 9y ago
      I love qtorrent. They even have built in torrents search for various sites like TPB and Linux isos ;)
    - lmm 9y ago
Klathmon 9y ago
I have to admit, BitTorrent is one of the things I took for granted. I never really thought about the details of how it works, or the really really impressive feats that were accomplished to get it to work. I knew it was a really good technology, but reading this and the comments here puts it on a whole other level. Why isn't this technology talked about more? Why are blockchains the big "thing" right now with people trying to use them everywhere to see where they fit best, but torrent networks are kind of just... ignored? The decentralized nature of it seems to open so many possibilities at first glance, is there a reason they aren't being taken advantage of? Is there some kind of "great filter" kind of thing that is preventing widespread usage of something like a torrent network?
- wongarsu 9y ago
  BitTorrent is "just" exchanging files via a p2p connection. It's kind of useful, a lot of projects use it in one way or another, but it's unlikely to be instrumental for "the next big thing". The BitTorrent DHT is great for storing and exchanging metadata, but a DHT is not something most people associate with BitTorrent (Bitcoin also has uses a DHT (for client discovery), as do countless other services). Blockchain technology on the other hand offers verifiable distributed timestamping (with ok-ish resolution). That has much wider applicability than just payment tracking (which is essentially all bitcoin does), which is why there's plenty of people exploring what's possible.
- tantalor 9y ago
  What is this "great filter"?
  - Klathmon 9y ago
    Its from the "fermi paradox"[0]. Basically it says that there might not be any other life out there because there is some kind of "event" or "limitation" of life that makes it so that it can only exist for so long, or it is just extremely difficult for life to get past this "filter" (and then there is the question of whether or not that filter is ahead of us, or behind us). In this case, I was trying to use it to ask if there is some kind of "unsolved problem", inherent limitation or issue/problem with torrent networks that prevents their widespread usage. [0] https://en.wikipedia.org/wiki/Great_Filter https://en.wikipedia.org/wiki/Great_Filter
redm 9y ago
I just don't see this technology ever going mainstream. I first deployed this type of application in 2003. It was named Redswoosh and did effectively the same thing as BitTorrent, just in a closed client. I was also a very early adopter of BitTorrent using it personally. Users hated it for general use, even when downloading big files. 1) They didn't like having to install/run some special software to download a file. 2) They didn't like the effects of uploading to others and it slowing down the connections. Consumer networks are asymmetric having far more download capacity in upload capacity. This makes sense since 1) most users download and want to use the available bandwidth for faster downloads, and 2) it prevents commercial applications on consumer circuits. This is far from ideal for applications like BitTorrent. I'm not saying there isn't an application for this technology, I'm saying all the good applications don't want to ask the users to pay for distribution to other users. Thus it's relegated to mostly piracy, open source, etc. Bittorrent Inc. has been trying to commercialize this for a decade now, I just don't see it happening. If there was anyone who could commercialize it, it was Travis Kalnik, and while he exited for 20m, he was very lucky, (and happy) to get out of that market.
- snakeanus 9y ago
  > I just don't see this technology ever going mainstream It already is though.

319 points by Nekit1234007 9y ago

lowglow 9y ago

Can someone diff the spec from the previous version? What's the changelog? :)

mouldysammich 9y ago
The main differences I can see is a change from SHA1 -> SHA2 and also seems to have added official spec for webtorrent.
- Luminarys 9y ago
  It also appears to be using a merkle hash tree for piece hashing now along with a few new peer wire messages to support that.
  - phire 9y ago
    It also switches to a merkle hash per-file and specifies that each file is aligned to the start of a piece, and the size of the last piece of each file matches the amount of remaining data. This means in large multi-file torrents you don't have to download (and store) the two extra 1-4mb pieces at the start/end of each file anymore.
- cjbprime 9y ago
  > official spec for webtorrent Huh, where do you see that? Not seeing any ctrl-f hits for webtorrent or webrtc.
  - mouldysammich 9y ago
    Whoops, i skimmed and they mentioned the end user web browser and I assumed they had. I guess this is why they say when you assume you make an ass out of you and me.
kpcyrd 9y ago
It would be interesting to see how the new version compares to ipfs.
- supergreg 9y ago
  The tree structure seems very similar. It would be nice if torrent clients could interact with ipfs or gain ipfs capabilities. Think torrents that update themselves when the files change (thanks to ipns).
  - sktrdie 9y ago
    There's already such BEP: http://bittorrent.org/beps/bep_0046.html http://bittorrent.org/beps/bep_0046.html - "Updating Torrents Via DHT Mutable Items"

jzelinskie 9y ago

This update to the spec is a modest change that's largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-byte length of a SHA1 checksum. They've decided to move forward with SHA256 truncated to 20 bytes to avoid incompatibilities with existing infrastructure such as the Mainline DHT. Beyond the hashing algorithm, some important additions that were previously proposals without widespread use (e.g. merkle tree for hashing pieces) are becoming required. The focus has mostly been on optimizing latency for the P2P protocol and making sane improvements to the file spec. I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Ideally, BitTorrent would be broken down into separate specifications that could be used together or in separate systems: one for the file format and piece representation for sharing files, one for the P2P protocol, and one for discovery (trackers, DHTs). I want to believe that there would be far more interesting P2P projects if you could just lift robust primitives from BitTorrent.

the8472 9y ago
The DHT BEPs specify a network that is only barely related to the bittorrent core protocol, they can already be used independently, and some people do. > I feel like trackers were largely overlooked in this update, but I'm biased because I work on a popular tracker. Yes, we did not pay much attention to trackers, but BEP52 basically seized the opportunity to do some incompatible changes we always wanted to do anyway (quite a few accumulated over the years), and there were no such open issues with the http tracker protocol.
- jzelinskie 9y ago
  >...and there were no such open issues with the http tracker protocol This is because the HTTP protocol is so much overhead that most trackers don't even really run it anymore. I think UDP being promoted to the spec would've been a step in the right direction. Modern trackers have a bunch of tricks like BEP34[0] to avoid getting pounded that would be great if every client conformed to. I hope I'm not coming off as aggressive. I really appreciate this work and I'm really glad to see a spec revision. It's just as you said, there's been many years and many good improvements that I'd like to see made while there's still a change to break things. [0]: http://www.bittorrent.org/beps/bep_0034.html http://www.bittorrent.org/beps/bep_0034.html

0x0 9y ago

What's the stuff about "proof layers", is that new in this v2? The paper briefly talks about proof layer requests. Is this something merkle-tree related? What is the purpose? Is it to prevent clients from lying about having pieces they do not have by requesting a verifiable random hash chunk?

the8472 9y ago
It's part of switching to merkle trees instead of flat piece lists. A merkle tree can only be verified if you either have a whole layer or send ancestor-siblings (uncle, great-uncle, etc.) along with a partial layer. Merkle trees allow torrents to start faster from magnet links since only the tree roots need to be front-loaded while the tree can be fetched incrementally.

shmerl 9y ago

Do all Bittorrent clients support it already?

silotis 9y ago
Currently no bittorrent clients support it. This is still just a draft. I've only just started working on an implementation for libtorrent, it will be quite a while before it is production ready.

richdougherty 9y ago

Rationale for hash function change: https://github.com/bittorrent/bittorrent.org/issues/58 https://github.com/bittorrent/bittorrent.org/issues/58 Discussion of other changes: https://github.com/bittorrent/bittorrent.org/pull/59 https://github.com/bittorrent/bittorrent.org/pull/59

Scaevolus 9y ago

1) Chunks don't span files. Each file is validated by the hash of its merkle tree. This is the biggest user-visible change, since it means you can download one file without downloading others. 2) SHA1 is replaced with SHA2-256 (2x longer hashes and not broken). 3) Files are represented by a tree structure instead of a list of dictionaries with paths-- this reduces duplication in deeply-nested hierarchies. 4) Backwards compatible-- you can make a .torrent file with both old and new pieces, and a swarm can speak either. This requires padding files from BEP47, which most clients probably don't support. Per-file metadata increases pretty significantly, from ~19B (just length) to ~68B (length + hash).

phire 9y ago
Per-file metadata increases significantly, but it gets rid of the per piece data (which in bittorrent v1 is 20 bytes of sha1 hash per piece and made up the bulk of the .torrent file). The .torrent file only stores the merkle tree's root hash for each file, and the torrent client will query it's peers to get the rest of the merkle tree (verifiable against the root hash). The leafs of the merkle tree are the hash of each 16kb block. Interesting consequences of this: Piece size isn't baked into the file anymore (and I've seen torrents with 16mb blocks), the client can dynamically chose it's verification piece size by requesting only so many layers of the merkle tree. Or it could skip requesting the tree and verify the whole file at once. Merkle tree roots will be globally unique. You can scan torrent files for duplicated files and download common files from multiple swarms.
- computerphage 9y ago
  > Or it could skip requesting the tree and verify the whole file at once. To clarify, this works by the client deterministically reconstructing the tree once they have the whole file, then checking the root's hash, correct?
  - phire 9y ago
    Yeah, it's a deterministic tree of hashes. Each leaf is the hash of a 16KB chunk. On the next layer up you have a series of nodes which are the hashes of the two leafs below it hashed together. You add enough layers until you get a single root hash at the root of the tree.

smegel 9y ago

Pity we will never see a genuine version of uTorrent that will support it. That was a real loss.

vanderZwan 9y ago
We have plenty of good open source alternatives now. qTorrent works fine
- smegel 9y ago
  Thanks I haven't hear of that. And it is written in C++ which is nice. Is it considered the spiritual successor to the original uTorrent?
  - nyolfen 9y ago
    it is: >The qBittorrent project aims to provide an open-source software alternative to µTorrent. though in my experience it is more of a memory hog and buggier than utorrent. but that doesn't stop me from using it
    - lucasverra 9y ago
      I've switched to transmission client and never looked back (4 yrs ago)
    - phire 9y ago
      It's nowhere near as perfect as utorrent once was. But it's pretty good and I have plenty of extra memory (it's currently using 305mb while seeding 40+ torrents, dropping down to 65mb if I pause them all.)
  - vopi 9y ago
    I love qtorrent. They even have built in torrents search for various sites like TPB and Linux isos ;)
  - lmm 9y ago

Klathmon 9y ago

I have to admit, BitTorrent is one of the things I took for granted. I never really thought about the details of how it works, or the really really impressive feats that were accomplished to get it to work. I knew it was a really good technology, but reading this and the comments here puts it on a whole other level. Why isn't this technology talked about more? Why are blockchains the big "thing" right now with people trying to use them everywhere to see where they fit best, but torrent networks are kind of just... ignored? The decentralized nature of it seems to open so many possibilities at first glance, is there a reason they aren't being taken advantage of? Is there some kind of "great filter" kind of thing that is preventing widespread usage of something like a torrent network?

wongarsu 9y ago
BitTorrent is "just" exchanging files via a p2p connection. It's kind of useful, a lot of projects use it in one way or another, but it's unlikely to be instrumental for "the next big thing". The BitTorrent DHT is great for storing and exchanging metadata, but a DHT is not something most people associate with BitTorrent (Bitcoin also has uses a DHT (for client discovery), as do countless other services). Blockchain technology on the other hand offers verifiable distributed timestamping (with ok-ish resolution). That has much wider applicability than just payment tracking (which is essentially all bitcoin does), which is why there's plenty of people exploring what's possible.
tantalor 9y ago
What is this "great filter"?
- Klathmon 9y ago
  Its from the "fermi paradox"[0]. Basically it says that there might not be any other life out there because there is some kind of "event" or "limitation" of life that makes it so that it can only exist for so long, or it is just extremely difficult for life to get past this "filter" (and then there is the question of whether or not that filter is ahead of us, or behind us). In this case, I was trying to use it to ask if there is some kind of "unsolved problem", inherent limitation or issue/problem with torrent networks that prevents their widespread usage. [0] https://en.wikipedia.org/wiki/Great_Filter https://en.wikipedia.org/wiki/Great_Filter

redm 9y ago

I just don't see this technology ever going mainstream. I first deployed this type of application in 2003. It was named Redswoosh and did effectively the same thing as BitTorrent, just in a closed client. I was also a very early adopter of BitTorrent using it personally. Users hated it for general use, even when downloading big files. 1) They didn't like having to install/run some special software to download a file. 2) They didn't like the effects of uploading to others and it slowing down the connections. Consumer networks are asymmetric having far more download capacity in upload capacity. This makes sense since 1) most users download and want to use the available bandwidth for faster downloads, and 2) it prevents commercial applications on consumer circuits. This is far from ideal for applications like BitTorrent. I'm not saying there isn't an application for this technology, I'm saying all the good applications don't want to ask the users to pay for distribution to other users. Thus it's relegated to mostly piracy, open source, etc. Bittorrent Inc. has been trying to commercialize this for a decade now, I just don't see it happening. If there was anyone who could commercialize it, it was Travis Kalnik, and while he exited for 20m, he was very lucky, (and happy) to get out of that market.

snakeanus 9y ago
> I just don't see this technology ever going mainstream It already is though.