4 ms·

Hi! Sarp here, author of the Build your own BitTorrent challenge on Codecrafters. Some back story: After being laid off from my FAANG job, I found myself very

by sarp 3y ago

Hi! Sarp here, author of the Build your own BitTorrent challenge on Codecrafters. Some back story: After being laid off from my FAANG job, I found myself very unmotivated to go back. I started looking for interesting programming projects to revive my interest in coding. While nomading, I discovered Codecrafters on Nomadlist and really liked the push code to git and pass different stages interaction. The gamification helped me focus and projects allowed me to go deeper on software I used (SQLite, Git, Redis etc.). I even picked up a new language (Go) to do the challenges with. After completing all the challenges on the site, I ran out of things to do. This is when I decided to build a BitTorrent client which was one of the highly voted ideas on the site. I learned many new things by building a BitTorrent client: the BitTorrent protocol, how torrent files are structured, encoding issues, pipelining network requests, url encoding binary values, using Channels in Go etc. I’d love any feedback on the challenge. Also happy to answer any questions!

sarupbanskota 3y ago
Are we gonna get extensions to the challenge?
- blurbleblurble 3y ago
  Better yet, extensions to implement BitTorrent extensions? Such as the mutable torrent BEP and others? It'd be great if this spawned hundreds of top quality BitTorrent clients
  - sarupbanskota 3y ago
    Build your own Extension
- sarp 3y ago
  To limit the scope of the challenge, I had to leave out a lot of cool features you see in modern clients: magnet links, UDP trackers, DHT I'm planning to add them over time starting with magnet links which is the highest voted extension idea right now
anjel 3y ago
Its not uncommon for invite-only trackers to be very prescriptive about which torrent clients can and can't be used with their private tracker. Any ideas on how to overcome this obstacle to wider adoption?
- vincentkriek 3y ago
  Is this something you expect to use as your main torrent client?
- CamperBob2 3y ago
  How do they tell one client from another?
  - thedaly 3y ago
    User agents, fingerprinting, etc. There are certainly ways to mask your client, but these would be considered cheating by most private trackers and would be grounds for a ban.
    - paledot 3y ago
      There's a whole ton of bluster in the torrent community. Bittorrent is a simple protocol, to the point of being naive (and therefore a fun toy network app project). Clients identify themselves to the tracker by user agent; there's really nothing else to fingerprint against. Claims to the contrary are almost certainly bullshit to scare people out of editing their user agent. Clients also self-report the amount of data transferred. That's not great in a community that fetishizes share ratios. I've heard an op say "there's no excuse for having a ratio less than 1", which makes as much mathematical sense as the parent who told my (math) teacher friend "this is [private school], no student should be below average". You can theoretically verify upload/download numbers because the total amount uploaded in a swarm should equal the total amount downloaded, but there are all kinds of reasons why the numbers wouldn't match. Maybe a client lost connection and couldn't send its final announce. Maybe one client is sending bad data (I'm not sure how that is reported, might be implementation specific). And clients only send transfer total updates when they connect to the tracker to change status or request more peers, so every client will have a different degree of staleness. Even if you can tell that someone in a swarm is lying, who is your culprit? As long as they're not being egregious, there's no way to tell.
rrishi 3y ago
> After completing all the challenges on the site, I ran out of things to do. This is when I decided to build a BitTorrent client which was one of the highly voted ideas on the site. Are you employed by them now?
- sarp 3y ago
  I'm not an employee of Codecrafters. I worked with them for the BitTorrent challenge as an independent contractor.
  - ferfumarma 3y ago
    This is such a cool result. I would love to see the next step of creating a tool like Resilio sync that builds upon this work!
andnand 3y ago
How did you learn to build a BitTorrent client? I love the idea of codecrafters and books that walk you through building something but I always struggle if I don't have something to get me started.
- sarp 3y ago
  My starting point was searching for tutorials and asking ChatGPT to implement a torrent parser :) There are great blog posts [0] for building a BitTorrent client. Along the way, I referenced open-source implementations and the BitTorrent Protocol Specification as well [1]. [0] https://blog.jse.li/posts/torrent/ https://blog.jse.li/posts/torrent/ [1] https://www.bittorrent.org/beps/bep_0003.html https://www.bittorrent.org/beps/bep_0003.html
ollien 3y ago
I don't know if this is feedback on the challenge per-se, but I was a little saddened that I couldn't add dependencies to my `Cargo.toml`; I wanted to solve part of the bencode challenge using nom (perhaps overkill, but it was for fun), but I can't. If this is a concern of load/execution times on a remote builder, it would be cool if I had some way to run the testcases locally to avoid this concern
- rohitpaulk 3y ago
  We just shipped support for this last week! You should be able to edit the file now. We’ll remove the comments saying those files can’t be edited soon.
  - ollien 3y ago
    Oh! Awesome! I'll have to give this a shot, then :)
friend_and_foe 3y ago
I've actually just begun work on a project involving bittorrent, I don't think I will be implementing a client but maybe, I'm certainly going to have to implement generation of .torrent files and magnet links, so I've been learning a little bit about the protocol. Interesting to see this pop up and maybe I can use it to help me make headway.

by sarp 3y ago

sarupbanskota 3y ago

Are we gonna get extensions to the challenge?

blurbleblurble 3y ago
Better yet, extensions to implement BitTorrent extensions? Such as the mutable torrent BEP and others? It'd be great if this spawned hundreds of top quality BitTorrent clients
- sarupbanskota 3y ago
  Build your own Extension
sarp 3y ago
To limit the scope of the challenge, I had to leave out a lot of cool features you see in modern clients: magnet links, UDP trackers, DHT I'm planning to add them over time starting with magnet links which is the highest voted extension idea right now

anjel 3y ago

Its not uncommon for invite-only trackers to be very prescriptive about which torrent clients can and can't be used with their private tracker. Any ideas on how to overcome this obstacle to wider adoption?

vincentkriek 3y ago
Is this something you expect to use as your main torrent client?
CamperBob2 3y ago
How do they tell one client from another?
- thedaly 3y ago
  User agents, fingerprinting, etc. There are certainly ways to mask your client, but these would be considered cheating by most private trackers and would be grounds for a ban.
  - paledot 3y ago
    There's a whole ton of bluster in the torrent community. Bittorrent is a simple protocol, to the point of being naive (and therefore a fun toy network app project). Clients identify themselves to the tracker by user agent; there's really nothing else to fingerprint against. Claims to the contrary are almost certainly bullshit to scare people out of editing their user agent. Clients also self-report the amount of data transferred. That's not great in a community that fetishizes share ratios. I've heard an op say "there's no excuse for having a ratio less than 1", which makes as much mathematical sense as the parent who told my (math) teacher friend "this is [private school], no student should be below average". You can theoretically verify upload/download numbers because the total amount uploaded in a swarm should equal the total amount downloaded, but there are all kinds of reasons why the numbers wouldn't match. Maybe a client lost connection and couldn't send its final announce. Maybe one client is sending bad data (I'm not sure how that is reported, might be implementation specific). And clients only send transfer total updates when they connect to the tracker to change status or request more peers, so every client will have a different degree of staleness. Even if you can tell that someone in a swarm is lying, who is your culprit? As long as they're not being egregious, there's no way to tell.

rrishi 3y ago

> After completing all the challenges on the site, I ran out of things to do. This is when I decided to build a BitTorrent client which was one of the highly voted ideas on the site. Are you employed by them now?

sarp 3y ago
I'm not an employee of Codecrafters. I worked with them for the BitTorrent challenge as an independent contractor.
- ferfumarma 3y ago
  This is such a cool result. I would love to see the next step of creating a tool like Resilio sync that builds upon this work!

andnand 3y ago

How did you learn to build a BitTorrent client? I love the idea of codecrafters and books that walk you through building something but I always struggle if I don't have something to get me started.

sarp 3y ago
My starting point was searching for tutorials and asking ChatGPT to implement a torrent parser :) There are great blog posts [0] for building a BitTorrent client. Along the way, I referenced open-source implementations and the BitTorrent Protocol Specification as well [1]. [0] https://blog.jse.li/posts/torrent/ https://blog.jse.li/posts/torrent/ [1] https://www.bittorrent.org/beps/bep_0003.html https://www.bittorrent.org/beps/bep_0003.html

ollien 3y ago

I don't know if this is feedback on the challenge per-se, but I was a little saddened that I couldn't add dependencies to my `Cargo.toml`; I wanted to solve part of the bencode challenge using nom (perhaps overkill, but it was for fun), but I can't. If this is a concern of load/execution times on a remote builder, it would be cool if I had some way to run the testcases locally to avoid this concern

rohitpaulk 3y ago
We just shipped support for this last week! You should be able to edit the file now. We’ll remove the comments saying those files can’t be edited soon.
- ollien 3y ago
  Oh! Awesome! I'll have to give this a shot, then :)

friend_and_foe 3y ago

I've actually just begun work on a project involving bittorrent, I don't think I will be implementing a client but maybe, I'm certainly going to have to implement generation of .torrent files and magnet links, so I've been learning a little bit about the protocol. Interesting to see this pop up and maybe I can use it to help me make headway.