10 ms·

Building a BitTorrent client from the ground up in Go

692 points by eat_veggies 7y ago

throwaway42534 7y ago
In the same vein, I started to implement a BitTorrent client in OCaml. https://github.com/phlalx/sawadee https://github.com/phlalx/sawadee Initially it was a project to learn about the Core/Async libraries. I will put it back to life at some point, but it grew quite big and became very time consuming for a single developer.
eat_veggies 7y ago
Over the holidays, I challenged myself to learn Go by torrenting the Debian ISO -- from scratch. This post is a bit of a brain dump about everything I've learned over the past week.
- giancarlostoro 7y ago
  > This isn’t the full story > For brevity, I included only a few of the important snippets of code. Notably, I left out all the glue code, parsing, unit tests, and the boring parts that build character. View my full implementation if you’re interested. Any chance you'll do independent write ups to cover all the missing bits? Especially the ones that build character. :)
  - devnullbyte 7y ago
    Would also be good to include how BT uses a merkel tree, that's quite a key component in the software.
    - smabie 7y ago
      Does any client support the new standard? I was under the impression that no one did.
    - anacrolix 7y ago
      It doesn't until the second version, which isn't mainstream.
      - anacrolix 7y ago
        https://github.com/anacrolix/torrent/issues/175#issuecomment-323524063 https://github.com/anacrolix/torrent/issues/175#issuecomment...
- mixmastamyk 7y ago
  Wow, I've been looking for a BT tutorial at about this level of abstraction for two years or so. And would have liked to have read it a decade or two back as well. Wondering if there are any similar docs or a book for implementing a server in Python, and a client in Python or Dart/Flutter? I know libtorrent bindings are a thing but the docs seemed quite dense and I didn't even know where to start—until now that is.
  - ngcc_hk 7y ago
    Same same. But just in Javascript more.
  - peterkelly 7y ago
    Here's a pretty good one in JavaScript: https://allenkim67.github.io/programming/2016/05/04/how-to-make-your-own-bittorrent-client.html https://allenkim67.github.io/programming/2016/05/04/how-to-m...
3fe9a03ccd14ca5 7y ago
Excellent write up and very readable code.
devadvance 7y ago
Awesome write-up and a great way to dig into the inner-workings of BitTorrent. Nice work! Glancing at the wiki page for BitTorrent[0], I'm a bit surprised that there isn't more of an effort to create cross-platform libraries and clients using Go or Rust for this. Seems like a perfect use case? For example, synapse [1]. [0] https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients#Libraries https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clien... [1] https://github.com/Luminarys/synapse https://github.com/Luminarys/synapse
- eat_veggies 7y ago
  I think BitTorrent is a very 2000's/early 2010's technology. That's when it feels like it peaked, at least. The future is probably more toward WebTorrents [0] (with WebRTC) and IPFS [1]. And those have great cross platform libraries. [0] https://webtorrent.io/ https://webtorrent.io/ [1] https://ipfs.io/ https://ipfs.io/
  - totalperspectiv 7y ago
    As a torrent newb, what's the tldr on on why webtorrent over BitTorrent?
    - eat_veggies 7y ago
      This is totally just my opinion, but the days of the mainstream consumer having a dedicated torrent client installed on their desktop computer (or even as a phone app) are pretty much over. But as a streaming technology, torrents are still pretty great. And like it or not, being able to run in browsers is a major advantage right now.
      - 3fe9a03ccd14ca5 7y ago
        This is true for the streaming use case, where you have a browser open and you want to watch a video. The case the author is going after is downloading a large Debian distro, so in the use-case of large, distributed file distribution there will always be solutions like the author.
diehunde 7y ago
Nice! Looks like a good idea to implement something from scratch to improve the understanding of a language and also a tool. Any other ideas about stuff to implement? Thanks.
- BookPage 7y ago
  I noticed my terminal emulator (Terminator) is a python app. I now have a todo item to try and write my own!
- eat_veggies 7y ago
  I built an http server and a markov chain library to learn python. They're both pretty worthwhile weekend-size projects. I heard cryptopals [0] is really good for learning both crypto and a new programming language. It might be cool to build things like a DNS resolver, an X window manager, gameboy game, or something from coreutils like grep or tar [0] https://cryptopals.com/ https://cryptopals.com/
  - jsjohnst 7y ago
    > It might be cool to build things like a DNS resolver I wouldn’t recommend writing a recursive resolver as a weekend project, to many edge cases that you’ll get frustrated by, but that said, I do use writing an authoritative DNS server as a great way to fully learn a new language. You get file handling, grammar parsing, serialization/deserialization, bitpacking, network I/O, data structures, both tcp and UDP socket handling, deamonization, concurrency, etc etc. Covers most of the more difficult parts of many languages and once you’ve done it once or twice, doing it again for new languages is less about figuring out how to do it, but rather how to do it in X language. At last count, I think I’ve done upwards of twenty languages now.
  - diehunde 7y ago
    Nice! Crypto stuff indeed sounds interesting.
- kccqzy 7y ago
  I implemented a blockchain-based cryptocurrency for fun. It sounds cliche but the details are surprisingly complicated: what exactly do you use as the criteria to determine whether you should accept a block? What are the ways a malicious actor can try to send you an invalid block? Just how exactly do you confirm a double-spending attempt? How do you pick a block based on which to mine a new block? Etc. https://github.com/kccqzy/SimpleBlockchain https://github.com/kccqzy/SimpleBlockchain
userbinator 7y ago
Strictly leeches (does not support uploading pieces) Be careful with using this client --- the swarm for a Linux ISO is probably a bit more forgiving, but you may get banned very quickly by the tracker or other clients, because they will definitely notice that.
war1025 7y ago
A decade ago now, I wrote a bit torrent client [1] that started as a project for a software development class. I think the focus of the class was roadmaps and other parts of the development process for a larger project. Most of the groups picked games or things like that. We chose to build a bit torrent client and I ended up continuing working on it for a year or so. I was pretty proud of it at the time. I'm sure there are a bunch of implementation details that seemed like a great idea as a Sophomore in college that no longer sound so clever, but the implementation is pretty sound. I used it to download a lot of stuff over the years. The part I think is most interesting is the code for managing the torrent pieces and writing them into the proper places for the various files within the torrent. (Not necessarily my implementation of it, just the math behind the process) [2] I remember feeling really clever the first time everything came together and actually downloaded a full torrent. My first bit software development victory. [1] https://github.com/war1025/Torrent https://github.com/war1025/Torrent [2] https://github.com/war1025/Torrent/blob/piecereg/tcl/tm/torrent/file/impl/FileAccessManagerImpl.java https://github.com/war1025/Torrent/blob/piecereg/tcl/tm/torr...
- ScarZy 7y ago
  Chapeau!
- userbinator 7y ago
  I used it to download a lot of stuff over the years. That's great to hear --- I know there are quite a few developers who would do lots of different projects, but only to the point of getting something that barely works, and then never use it again. Having it in regular use means you have to continue to fix bugs and make improvements, which a lot of developers unfortunately seem to hate doing, although it's often the more valuable skill. "A true craftsman makes his own tools," as the old saying goes.
- kccqzy 7y ago
  > The part I think is most interesting is the code for managing the torrent pieces and writing them into the proper places for the various files within the torrent. (Not necessarily my implementation of it, just the math behind the process) Can you explain a little bit how this is interesting mathematically? On first thought, it seems like you would probably use a lot of pwrite, or if want convenience over safety, mmap all the files and write directly to memory. Not to diminish your achievement, but I didn't immediately see anything interesting here. Can you explain a bit more?
jamra 7y ago
I noticed a few things with the Go code that confused me. I haven't coded in Go in quite some time, so I may be way off. `copy(buf[1:], h.Pstr)` In this line, are you copying the entire buffer to a string? Doesn't it overflow into other data elements? Also, and I may be wrong, in the following line, it appears that you are casting to a []byte when it's already a slice of bytes, which should still be fine. `peers[i].Port = binary.BigEndian.Uint16([]byte(peersBin[offset+4 : offset+6])` I really enjoyed the tone and the code. I'm not done with the article, but I love it so far.
- eat_veggies 7y ago
  thank you for the feedback! It's really helpful. copy's arguments are like copy(destination, source) so we're copying the string into the buffer. Also, copy will never overflow because it will only copy up to the length of the shortest buffer. you're right about the unnecessary cast to []byte -- that function used to take a string as arguments, and when I changed it, I didn't change the rest of it. I've removed it.
plupopit 7y ago
Nice post. But why this emphasis on bittorents illegal use cases in the beginning? It's just a protocol and that's it. What people use it for is diverse and something different. In a post about a new car nobody would even mention it's possible use in illegal steet races.
ngcc_hk 7y ago
Can it be done in node.js under a VM so to isolate the traffic and force it to use vpn etc.
- ngcc_hk 7y ago
  Found it based on comments here: https://github.com/webtorrent/webtorrent https://github.com/webtorrent/webtorrent I guess the VM part is just to use a unikernel then try to find a way to auto use vpn.
- deanclatworthy 7y ago
  Use proxychains for this
_tkzm 7y ago
i wanted to do the same not that long ago but i abondoned that idea when i could not find UI library i would like to work with and i was not looking to make a cli application.
jbverschoor 7y ago
Great write up, very clear explanation about the torrent protocol, and nice to see the rational behind certain code decisions.
ignoramous 7y ago
The blog doesn't discuss it, but the choice of DHT implementation in BitTorrent is Kademlia [0] whilst Chord is more popular, used by Amazon Dynamo and Facebook Cassandra [1]. One of the authors of Kademlia, David Mazières, later co-authored the Stellar crypto-currency protocol [2] with Jed McCaleb of Ripple fame. That said, quite famously, Bram Cohen, the co-inventor of BitTorrent, failed the Google interviews. [0] https://news.ycombinator.com/item?id=18711980 https://news.ycombinator.com/item?id=18711980 [1] https://news.ycombinator.com/item?id=3480480 https://news.ycombinator.com/item?id=3480480 [2] https://news.ycombinator.com/item?id=16125920 https://news.ycombinator.com/item?id=16125920
- kodablah 7y ago
  Kademlia, especially with pieces of SKademlia, is much better with untrusted peers and is much more popular in p2p projects such as BT, I2P, Ethereum, IPFS, etc.
- cristaloleg 7y ago
  Any sources regarding failed interview? Googles (ha-ha) says nothing.
  - ignoramous 7y ago
    May be the down-voters think I'm a troll? I thought it was an interesting note about the creator of a protocol responsible for a third of all Internet traffic. Or, does it highlight the alleged ineffectiveness of FAANG-style tech interviews? Btw, here's a ref: https://web.archive.org/web/20200105225938/https://www.theregister.co.uk/2007/01/05/google_interview_tales/ https://web.archive.org/web/20200105225938/https://www.there...
ovebepari 7y ago
Any resources for doing the same thing in python?
jijji 7y ago
I played with this it requires go 1.13 and seems to time out and close if one of the trackers is unresponsive for more than 10 seconds. It has no command line options and crashes if no options are present. It also doesnt support magnet links, only torrent files.
tgrzinic 7y ago
@eat_veggies The writeup is very useful. Can you describe the process of learning two concepts simultaneously? Or you knew before the parts of Bitorrent and how to implement it in some other language?

692 points by eat_veggies 7y ago

In the same vein, I started to implement a BitTorrent client in OCaml. https://github.com/phlalx/sawadee https://github.com/phlalx/sawadee Initially it was a project to learn about the Core/Async libraries. I will put it back to life at some point, but it grew quite big and became very time consuming for a single developer.

eat_veggies 7y ago

Over the holidays, I challenged myself to learn Go by torrenting the Debian ISO -- from scratch. This post is a bit of a brain dump about everything I've learned over the past week.

giancarlostoro 7y ago
> This isn’t the full story > For brevity, I included only a few of the important snippets of code. Notably, I left out all the glue code, parsing, unit tests, and the boring parts that build character. View my full implementation if you’re interested. Any chance you'll do independent write ups to cover all the missing bits? Especially the ones that build character. :)
- devnullbyte 7y ago
  Would also be good to include how BT uses a merkel tree, that's quite a key component in the software.
  - smabie 7y ago
    Does any client support the new standard? I was under the impression that no one did.
  - anacrolix 7y ago
    It doesn't until the second version, which isn't mainstream.
    - anacrolix 7y ago
      https://github.com/anacrolix/torrent/issues/175#issuecomment-323524063 https://github.com/anacrolix/torrent/issues/175#issuecomment...
mixmastamyk 7y ago
Wow, I've been looking for a BT tutorial at about this level of abstraction for two years or so. And would have liked to have read it a decade or two back as well. Wondering if there are any similar docs or a book for implementing a server in Python, and a client in Python or Dart/Flutter? I know libtorrent bindings are a thing but the docs seemed quite dense and I didn't even know where to start—until now that is.
- ngcc_hk 7y ago
  Same same. But just in Javascript more.
- peterkelly 7y ago
  Here's a pretty good one in JavaScript: https://allenkim67.github.io/programming/2016/05/04/how-to-make-your-own-bittorrent-client.html https://allenkim67.github.io/programming/2016/05/04/how-to-m...

3fe9a03ccd14ca5 7y ago

Excellent write up and very readable code.

devadvance 7y ago

Awesome write-up and a great way to dig into the inner-workings of BitTorrent. Nice work! Glancing at the wiki page for BitTorrent[0], I'm a bit surprised that there isn't more of an effort to create cross-platform libraries and clients using Go or Rust for this. Seems like a perfect use case? For example, synapse [1]. [0] https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients#Libraries https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clien... [1] https://github.com/Luminarys/synapse https://github.com/Luminarys/synapse

eat_veggies 7y ago
I think BitTorrent is a very 2000's/early 2010's technology. That's when it feels like it peaked, at least. The future is probably more toward WebTorrents [0] (with WebRTC) and IPFS [1]. And those have great cross platform libraries. [0] https://webtorrent.io/ https://webtorrent.io/ [1] https://ipfs.io/ https://ipfs.io/
- totalperspectiv 7y ago
  As a torrent newb, what's the tldr on on why webtorrent over BitTorrent?
  - eat_veggies 7y ago
    This is totally just my opinion, but the days of the mainstream consumer having a dedicated torrent client installed on their desktop computer (or even as a phone app) are pretty much over. But as a streaming technology, torrents are still pretty great. And like it or not, being able to run in browsers is a major advantage right now.
    - 3fe9a03ccd14ca5 7y ago
      This is true for the streaming use case, where you have a browser open and you want to watch a video. The case the author is going after is downloading a large Debian distro, so in the use-case of large, distributed file distribution there will always be solutions like the author.

diehunde 7y ago

Nice! Looks like a good idea to implement something from scratch to improve the understanding of a language and also a tool. Any other ideas about stuff to implement? Thanks.

BookPage 7y ago
I noticed my terminal emulator (Terminator) is a python app. I now have a todo item to try and write my own!
eat_veggies 7y ago
I built an http server and a markov chain library to learn python. They're both pretty worthwhile weekend-size projects. I heard cryptopals [0] is really good for learning both crypto and a new programming language. It might be cool to build things like a DNS resolver, an X window manager, gameboy game, or something from coreutils like grep or tar [0] https://cryptopals.com/ https://cryptopals.com/
- jsjohnst 7y ago
  > It might be cool to build things like a DNS resolver I wouldn’t recommend writing a recursive resolver as a weekend project, to many edge cases that you’ll get frustrated by, but that said, I do use writing an authoritative DNS server as a great way to fully learn a new language. You get file handling, grammar parsing, serialization/deserialization, bitpacking, network I/O, data structures, both tcp and UDP socket handling, deamonization, concurrency, etc etc. Covers most of the more difficult parts of many languages and once you’ve done it once or twice, doing it again for new languages is less about figuring out how to do it, but rather how to do it in X language. At last count, I think I’ve done upwards of twenty languages now.
- diehunde 7y ago
  Nice! Crypto stuff indeed sounds interesting.
kccqzy 7y ago
I implemented a blockchain-based cryptocurrency for fun. It sounds cliche but the details are surprisingly complicated: what exactly do you use as the criteria to determine whether you should accept a block? What are the ways a malicious actor can try to send you an invalid block? Just how exactly do you confirm a double-spending attempt? How do you pick a block based on which to mine a new block? Etc. https://github.com/kccqzy/SimpleBlockchain https://github.com/kccqzy/SimpleBlockchain

userbinator 7y ago

Strictly leeches (does not support uploading pieces) Be careful with using this client --- the swarm for a Linux ISO is probably a bit more forgiving, but you may get banned very quickly by the tracker or other clients, because they will definitely notice that.

war1025 7y ago

A decade ago now, I wrote a bit torrent client [1] that started as a project for a software development class. I think the focus of the class was roadmaps and other parts of the development process for a larger project. Most of the groups picked games or things like that. We chose to build a bit torrent client and I ended up continuing working on it for a year or so. I was pretty proud of it at the time. I'm sure there are a bunch of implementation details that seemed like a great idea as a Sophomore in college that no longer sound so clever, but the implementation is pretty sound. I used it to download a lot of stuff over the years. The part I think is most interesting is the code for managing the torrent pieces and writing them into the proper places for the various files within the torrent. (Not necessarily my implementation of it, just the math behind the process) [2] I remember feeling really clever the first time everything came together and actually downloaded a full torrent. My first bit software development victory. [1] https://github.com/war1025/Torrent https://github.com/war1025/Torrent [2] https://github.com/war1025/Torrent/blob/piecereg/tcl/tm/torrent/file/impl/FileAccessManagerImpl.java https://github.com/war1025/Torrent/blob/piecereg/tcl/tm/torr...

ScarZy 7y ago
Chapeau!
userbinator 7y ago
I used it to download a lot of stuff over the years. That's great to hear --- I know there are quite a few developers who would do lots of different projects, but only to the point of getting something that barely works, and then never use it again. Having it in regular use means you have to continue to fix bugs and make improvements, which a lot of developers unfortunately seem to hate doing, although it's often the more valuable skill. "A true craftsman makes his own tools," as the old saying goes.
kccqzy 7y ago
> The part I think is most interesting is the code for managing the torrent pieces and writing them into the proper places for the various files within the torrent. (Not necessarily my implementation of it, just the math behind the process) Can you explain a little bit how this is interesting mathematically? On first thought, it seems like you would probably use a lot of pwrite, or if want convenience over safety, mmap all the files and write directly to memory. Not to diminish your achievement, but I didn't immediately see anything interesting here. Can you explain a bit more?

jamra 7y ago

I noticed a few things with the Go code that confused me. I haven't coded in Go in quite some time, so I may be way off. `copy(buf[1:], h.Pstr)` In this line, are you copying the entire buffer to a string? Doesn't it overflow into other data elements? Also, and I may be wrong, in the following line, it appears that you are casting to a []byte when it's already a slice of bytes, which should still be fine. `peers[i].Port = binary.BigEndian.Uint16([]byte(peersBin[offset+4 : offset+6])` I really enjoyed the tone and the code. I'm not done with the article, but I love it so far.

eat_veggies 7y ago
thank you for the feedback! It's really helpful. copy's arguments are like copy(destination, source) so we're copying the string into the buffer. Also, copy will never overflow because it will only copy up to the length of the shortest buffer. you're right about the unnecessary cast to []byte -- that function used to take a string as arguments, and when I changed it, I didn't change the rest of it. I've removed it.

plupopit 7y ago

Nice post. But why this emphasis on bittorents illegal use cases in the beginning? It's just a protocol and that's it. What people use it for is diverse and something different. In a post about a new car nobody would even mention it's possible use in illegal steet races.

ngcc_hk 7y ago

Can it be done in node.js under a VM so to isolate the traffic and force it to use vpn etc.

ngcc_hk 7y ago
Found it based on comments here: https://github.com/webtorrent/webtorrent https://github.com/webtorrent/webtorrent I guess the VM part is just to use a unikernel then try to find a way to auto use vpn.
deanclatworthy 7y ago
Use proxychains for this

_tkzm 7y ago

i wanted to do the same not that long ago but i abondoned that idea when i could not find UI library i would like to work with and i was not looking to make a cli application.

jbverschoor 7y ago

Great write up, very clear explanation about the torrent protocol, and nice to see the rational behind certain code decisions.

ignoramous 7y ago

The blog doesn't discuss it, but the choice of DHT implementation in BitTorrent is Kademlia [0] whilst Chord is more popular, used by Amazon Dynamo and Facebook Cassandra [1]. One of the authors of Kademlia, David Mazières, later co-authored the Stellar crypto-currency protocol [2] with Jed McCaleb of Ripple fame. That said, quite famously, Bram Cohen, the co-inventor of BitTorrent, failed the Google interviews. [0] https://news.ycombinator.com/item?id=18711980 https://news.ycombinator.com/item?id=18711980 [1] https://news.ycombinator.com/item?id=3480480 https://news.ycombinator.com/item?id=3480480 [2] https://news.ycombinator.com/item?id=16125920 https://news.ycombinator.com/item?id=16125920

kodablah 7y ago
Kademlia, especially with pieces of SKademlia, is much better with untrusted peers and is much more popular in p2p projects such as BT, I2P, Ethereum, IPFS, etc.
cristaloleg 7y ago
Any sources regarding failed interview? Googles (ha-ha) says nothing.
- ignoramous 7y ago
  May be the down-voters think I'm a troll? I thought it was an interesting note about the creator of a protocol responsible for a third of all Internet traffic. Or, does it highlight the alleged ineffectiveness of FAANG-style tech interviews? Btw, here's a ref: https://web.archive.org/web/20200105225938/https://www.theregister.co.uk/2007/01/05/google_interview_tales/ https://web.archive.org/web/20200105225938/https://www.there...

ovebepari 7y ago

Any resources for doing the same thing in python?

jijji 7y ago

I played with this it requires go 1.13 and seems to time out and close if one of the trackers is unresponsive for more than 10 seconds. It has no command line options and crashes if no options are present. It also doesnt support magnet links, only torrent files.

tgrzinic 7y ago

@eat_veggies The writeup is very useful. Can you describe the process of learning two concepts simultaneously? Or you knew before the parts of Bitorrent and how to implement it in some other language?