6 ms·
Building a BitTorrent client from the ground up in Go (2020)
- unboxingelf 4y agoAs someone who writes Go code daily, nice Go code!
- DoctorOW 4y agoOne of my favorite things about Go's language design is that difference between really good code and average-level code is much smaller.
- adastra22 4y agoThanks, this is really cool. I don't have anything to add; I just want to let the author know this is appreciated. I recently decided to learn Go, and articles like this are a great way to get a feel for how Go programmers think about solving problems with the tools available to them. The kind of knowledge you don't get from just skimming The Go Programming Language and such.
- quirino 4y agoI've recently started learning Go and chose a project-based approach. I've been reading "Writing an Interpreter in Go" and "Powerful Command-Line Applications in Go" and it's been loads of fun.
- adastra22 4y agoThanks for the recommendations!
- x86hacker1010 4y agoCheck out distributed services in go as well. Best technical book I’ve ever read
- mdaniel 4y agoThe project author's previous submission: https://news.ycombinator.com/item?id=21958359 https://news.ycombinator.com/item?id=21958359
- andirk 4y agoThey're big fans of bittorrent, it appears.
- aliqot 4y agoBit Torrent is a cornerstone of the modern internet.
- andirk 4y agoAll the web3 talk is scoffed at by many, but I believe in decentralizing the internet as much as possible and am a big fan of Bittorrent. To quote Satoshi Nakamoto, "For transferable proof of work tokens to have value, they must have monetary value. To have monetary value, they must be transferred within a very large network – for example a file trading network akin to bittorrent."
- mdaniel 4y agoI don't believe BT will truly take off until web seeding is functional in all clients and I can use (a site | series of sites) to make a torrent of an existing thing. There was such a site (whose cute URL escapes me) where one could plug in a URL, it would download and cook a .torrent file but with the web seeds pointed at the source, and from that point forward others could get the cached version and the process was bootstrapped I was also saddened by AWS dropping torrent support from their S3 API because that's one more easy way that folks could provide torrent bootstrapping "for free"
- killingtime74 4y agoTake off? It's unfortunately waning. Used to be 35% of all internet traffic https://torrentfreak.com/bittorrent-is-still-the-king-of-upstream-internet-traffic-but-for-how-long-220304/ https://torrentfreak.com/bittorrent-is-still-the-king-of-ups...
- riley_dog 4y agoThis looks to have been written on Jan 4, 2020.
- latchkey 4y agoPreviously: https://news.ycombinator.com/item?id=21958359 https://news.ycombinator.com/item?id=21958359
- throwaway894345 4y agoThis didn’t dive into it, but I appreciated the suggested search terms for information on decentralized peer discovery. I always wondered how that would work, especially the bootstrapping process. Ultimately I found this StackOverflow answer to be really helpful: https://stackoverflow.com/a/22240583 https://stackoverflow.com/a/22240583
- banana_giraffe 4y agoOne of the things that made the bootstrapping click for me was realizing there are in fact servers that DHT clients talk to once to find an entrance to the network. They all have a list (such as https://github.com/qbittorrent/qBittorrent/blob/c80238d66ff3d554abab662a2b208cc901d3ee0e/src/base/bittorrent/sessionimpl.cpp#L1811 https://github.com/qbittorrent/qBittorrent/blob/c80238d66ff3... )
- throaway53dh 4y agoWhy is it so hard to in block torrent if all they have to d ok is block these DHT servers
- 5e92cb50239222b 4y agoBlocking DHT bootstrap nodes only takes care of DHT. This is only one way of obtaining peers, so even if it's not available, you can still fetch peer information from torrent trackers and receive potential peer addresses from other nodes through Peer Exchange. DHT is required when your torrent file does not contain any tracker URLs (and you didn't add any yourself). In this case if it's blocked, then you have a problem. Private trackers typically disable DHT for all torrents (so their torrents don't leak to outside users), along with PEX. It's just a flag in the torrent file, so you can force-enable it if you want (but it won't help much since the vast majority of other peers will probably abide by the rules).
- slim 4y agoThose are used only by new users the first time they use the app. They are regularly blocked, or retired (because maintainer decided so), it does not affect the network because they have nothing special. New ips are added to the list. You could become one of those simply by letting your computer online 24/7.
- chrsig 4y agoGreat post One nit, with the font: l4:spami7ee the first two characters look an awful lot like the number "14", when they're actually "L4". I'm not sure if bencoding is case sensitive or if anohter font may better differentiate the two, but that may be something the author may want to poke at if they happen to read this comment :)
- fazfq 4y agoThe l looks taller than the 4 to me, which makes it clear it's not another number
- ben_ja_min 4y agoLooks like a one to my bad eyes.
- pianoben 4y agoI had the same confusion! In my browser, with the default monospace font, the two characters are so similarly as to be indistinguishable. Reading the next example, showing an encoded dictionary, was what made the penny drop. Bencode doesn't go out of its way to be visually accessible, so different font choices would help.
- palata 4y agoIt is only downloading, right? Is that tolerated? Anyway it's a cool post! Would be nice to see a follow-up for the upload/sharing part!
- mdaniel 4y agoYou may be thinking of private trackers, where "ratio" is the currency of participation in that community. Debian, Libre Office, and even Humble Bundle care more about the community of seeders sharing the bandwidth load than whether anyone seeds their ratio
- muwtyhg 4y agoI believe they are trying to clarify if this code is only for the 'downloading' portion of a BitTorrent client. It does not seem have any code for handling requests for pieces coming from other peers.
- palata 4y agoNo. I'm thinking that a BitTorrent client, in my naive understanding, should share the file while it downloads it. Otherwise it's just a client/server system, isn't it? That code is only downloading, as far as I can tell. Never sharing the file. I was just asking if that's tolerated in BitTorrent. My guess was that it is not.
- mdaniel 4y agoIt is a client/server system, but pulling from $n hosts instead of just 1, spreading out the load and also downtime risk across all $n hosts. And the reason I mentioned the "ratio currency" is your use of the word "tolerated" here: BitTorrent isn't the mafia, it's just a protocol that offers the ability to share pieces back to the Internet. As another concrete example, aria2c (https://github.com/aria2/aria2#readme https://github.com/aria2/aria2#readme) behaves similarly: using BT to download, but then exiting without attempting to seed anything
- 4y ago
- odo1242 4y agonice cartoons! :D
- tommica 4y agoReally nice share! I think I'll have a follow-along with this next weekend to improve my go skills and try to learn rust by attempting to port this to it.
- jbirer 4y agoOne of the best posts lately. Thanks.
- hknmtt 4y agothis is good info. i was thinking about making my own, before i discovered picotorrent. but the lack of ui libraries for go prevented me from moving forward. nowadays, with nuts and others, it might be worth a shot.
- merpkz 4y ago> "They’ll send us an unchoke message to let us know that we can begin asking them for data." What is the purpose of that choke part when connecting to remote peers? Is that some kind of flood protection?
- nprateem 4y agoYou can normally control the number of peers you share with, so I guess if 50 peers are connected but you only want to share with 10, then when one completes you'd send an unchoke message to another peer to tell it you'll share with them. Meanwhile the other 39 will stay choked. I assume that's what's going on here.
- merpkz 4y agoThat makes sense, thanks. It just occurred to me how interesting of a challenge it is to write a bittorrent client. There are so much issues to work on, like how to effectively retrieve chunks of files and in meantime balance upload speed equally to remote clients of file parts you actually have already.
- amits1995 4y agowhat happens when the peer is behind NAT? you need to be able to use TCP Hole Punching
- klabb3 4y agoEven if the peer isn't behind a NAT (or on IPv6), you still need to pass the firewall, which won't let incoming packets through. This can be solved by simply connecting from both ends simultaneously, (first attempt fails but opens the fw, second attempt in the other direction succeeds) You only need hole punching if both peers are behind a NAT (TCP hole punching is substantially more complex and fragile than UDP). You can reduce the need for hole punching if you complicate the signaling protocol (eg "please dial me at <addr>"). That said, bittorrent is quite resilient to NATs in practice, because since it's many-to-many, chances are even NATed clients will find many non-NATed clients.
- 3ln00b 4y agoVery helpful. Can someone recommend more of these learn by doing programming blogs?
- elcapitan 4y agoThis was posted recently: https://github.com/codecrafters-io/build-your-own-x https://github.com/codecrafters-io/build-your-own-x
- UltraViolence 4y agoPersonally I would've preferred a Rust implementation, but nice article nonetheless.
- LinkLink 4y agoHey Jesse, we went to school together but I'm sure that's a ton of people you know by now ;). If you can figure it out hit me up ill show you some Zebra industry stuff I've reversed.