10 ms·
Building a BitTorrent client from scratch in C#
- voltagex_ 10y agoBEncoding and variants like REncoding are possibly one of my least favourite things ever. If you deal with the Deluge torrent client API you'll see it everywhere. That aside, fantastic work on this, I think previously the only Bittorrent library for C# was an abandoned Mono project.
- douche 10y agoYeah, monotorrent doesn't really work very effectively. Tends to get blocked by a lot of peers and really mediocre throughout when it isn't blocked. The last time I was trying to build something that used bittorrent, I fell back on launching aria2 and redirecting and parsing its stdout, which as crazy as it was, worked much better.
- Ardren 10y agoAria2 actually has a RPC interface which is pretty easy to work with (json or xml over http or web sockets). https://aria2.github.io/manual/en/html/aria2c.html#rpc-interface https://aria2.github.io/manual/en/html/aria2c.html#rpc-inter...
- masklinn 10y ago> BEncoding and variants like REncoding are possibly one of my least favourite things ever. Can't be worse than ASN.1 can it?
- orbitur 10y agoYes, it can be worse, since ASN.1 is relatively "popular" and there's a decent pool of knowledge.
- nickpsecurity 10y agoDepends on if ignoring a high-assurance implementation is bad: http://cps-vo.org/node/1577 http://cps-vo.org/node/1577 Already bad if you wanted ASN.1 but can't get Galois's version. If you can get it, then choosing an ASN.1 alternative can be bad since it probably won't have a formal spec, verified parser, Haskell implementation, and so on. Probably a drop in correctness in some corner case vs whatever they made. Note: I'd like to see them do a high-assurance JSON and/or XDR parser instead of just ASN.1. I know they did a Haskell-to-JSON library already. A strong one that extracted parsers or generators from a user-supplied specification with plugins for various programming languages would be nice.
- aashu_dwivedi 10y agoI wish there's a similar article in python.
- Razengan 10y agoI would love one, and other similarly meaty projects, in Swift 3.
- AsyncAwait 10y agoI am actually working on a GUI BitTorrent client in Swift, just converted my code to Swift 3 this morning, (I'm in the EU), quite far to go still, but I hope to have an alpha release out in Q4.
- masklinn 10y agoSkimming it, the article seems pretty detailed, it should be possible to follow it in other languages, or even to convert it to a polyglot guide.
- emodendroket 10y agoIsn't the original client a python program?
- spaam 10y agoyeah, you can still download it from http://web.archive.org/web/20110712221649/http://download.bittorrent.com/dl/archive/ http://web.archive.org/web/20110712221649/http://download.bi...
- mac01021 10y agoIt seems to me that the C# code listings here look basically just like the equivalent Python. I wouldn't let the name of the language stop you.
- romanvm 10y agoI guess a native Python implementation would be too slow. However, there is a fantastic libtorrent library that has Python bindings and allows to implement a torrent client in Python relatively easily. BTW, regarding the original article, there is also a MonoTorrent library for .NET. Despite the name it can be compiled by Visual Studio. The original library was abandoned a while ago and seems to be buggy, but I was able to make a very simple .NET client with WinForms UI using this fork: https://github.com/ErtyHackward/monotorrent https://github.com/ErtyHackward/monotorrent
- mafuy 10y agoIs the site down?
- cheatdeath 10y agoIt's not completely down, the server does seems to be struggling though.
- dethi 10y agoHere is the text-only version from Google Cache http://webcache.googleusercontent.com/search?q=cache:http://seanjoflynn.com/research/bittorrent.html&num=1&strip=1&vwsrc=0 http://webcache.googleusercontent.com/search?q=cache:http://...
- kyriakos 10y agonot really just takes minutes to load
- cheatdeath 10y agoIf anyone's having issues, I've mirrored it at https://cheatdeath.github.io/research-bittorrent-doc/ https://cheatdeath.github.io/research-bittorrent-doc/ edit: I'm the author, let me know if you have any questions.
- lossolo 10y agoGreat work. Does it work under .NET Core ?
- jsingleton 10y agoFrom a quick look at the source [0], it looks like it supports Mono. I can't see anything stopping it being ported to Core, apart from JonSkeet.MiscUtil may not support Core. If you package this up as a NuGet package and support Core then can you please ping me and I'll add it to https://anclafs.com https://anclafs.com. [0] https://github.com/cheatdeath/research-bittorrent https://github.com/cheatdeath/research-bittorrent
- nick_ 10y agoI've been prototyping a .NET Core port. JonSkeet.MiscUtil source code is fine for Core, but for the BitTorrent code we need to change the HttpWebRequest objects to HttpClient and rewrite the Begin/End/IAsyncResult/callback-style to async/await.
- jsingleton 10y agoFYI Jon has said to avoid using MiscUtil. https://github.com/jpsingleton/ANCLAFS/issues/6#issuecomment-231041947 https://github.com/jpsingleton/ANCLAFS/issues/6#issuecomment...
- cheatdeath 10y agoI'm not that familiar with .NET Core. It doesn't look like MiscUtil is compatible with it. The executable project has some minor references to Mono.Posix just so it can catch kill signals while running in the Terminal and die gracefully. Both dependencies can be removed quite easily from the project.
- whoisthemachine 10y agoNot sure if it's a part of the standard library, but the .Net variant of C# contains a sorted dictionary: https://msdn.microsoft.com/en-us/library/f7fta44c(v=vs.110).aspx https://msdn.microsoft.com/en-us/library/f7fta44c(v=vs.110)....
- whoisthemachine 10y agoAlso, fantastic work. The readability of C# really shines through here - I could grok all of this code instantly :)
- cheatdeath 10y agoThanks. Actually organising the write up forced me to tidy up the code much more than I otherwise would have. I definitely find C# to be one of the more readable languages although I have had to debug and untangle some C# messes before. Yeah I remember changing it to a SortedDictionary but I changed it back. I can't remember exactly why, possibly because it's supposed to be sorted by raw UTF8 bytes rather than a nice neat C# string and I didn't want to start using byte arrays for dictionary keys. I guess it only needs to be sorted when in the BEncoding format and it felt better to keep the internal structure as simple as possible. The tradeoff is it doesn't support incorrectly encoded torrent files – I'm really not sure how much of an issue that is.
- kidmenot 10y agoAgreed. Properly written C# code is very readable. I write most things in it these days and couldn't be happier about what this language has become. I have a few minor gripes with id, but they mainly relate to missing syntactic sugar rather than actual shortcomings. Kudos to OP, great job. EDIT: fix up grammar mistake.
- Uptrenda 10y agoIt's really great work OP. I know this would have taken you a long time to do but part of me can't help but wonder if programming is becoming even more like paint by numbers than it already is.
- allenkim6 10y agoIf anyone is interested in a similar writeup for node.js check out my tutorial here: http://allenkim67.github.io/bittorrent/2016/05/04/how-to-make-your-own-bittorrent-client.html http://allenkim67.github.io/bittorrent/2016/05/04/how-to-mak...
- finnn 10y agoHTTPS Everywhere redirects me to the HTTPS version of the page, but you've hard-coded http:// http:// links for some/all of the resources, which the browser refuses to load, so it just looks like some big gray boxes.
- allenkim6 10y agoHey thanks, it looks like the link for the css was the problem so I've just fixed that for now.
- ZanyProgrammer 10y agoSomewhat disappointing that it's just a console app. I'd love to be able to do cross platform C# desktop development. There shock be something equivalent to WinForms/WPF on OSs other than Windows.
- ohitsdom 10y agoThis is now possible with the new .NET Core. https://docs.microsoft.com/en-us/dotnet/articles/core/index https://docs.microsoft.com/en-us/dotnet/articles/core/index
- revelation 10y agoThe one thing they explicitly do not have on the roadmap for .NET Core is any sort of desktop environment / UI stuff. There won't be a Forms or WPF port.
- flamedoge 10y agoSure there won't be shared UI development support, but you can easily encapsulate app logic from UI. If you're brave enough you can try using Gtk# from http://www.mono-project.com/docs/gui/ http://www.mono-project.com/docs/gui/
- egeozcan 10y agoI got the impression that Xamarin was going in that direction?
- nbarbettini 10y agoXamarin is specifically for app and UI on Android/iOS/Windows Phone.
- michaeldwan 10y agoThey released Mac bindings for Xamarin Forms recently. Not a stretch to imagine unified UI for desktops.
- jdudek 10y agoHere’s a similar write-up on building a BitTorrent client in Haskell: https://blog.chaps.io/2015/10/05/torrent-client-in-haskell-1.html https://blog.chaps.io/2015/10/05/torrent-client-in-haskell-1...
- vishnuks 10y agoFor interested people there is a great write up on Tox protocol here https://toktok.github.io/spec https://toktok.github.io/spec
- mattcopp 10y agoGreat work! Quite annoying actually. I finished my own implementation in Python at about 10pm last night, this would have been most useful. I'm no C# coder, but it's nicely readable, and this is a much better write up than I'm sure I could do. If anyone who hasn't tried doing this before, the "official" BitTorrent spec docs, namely BEP-3 (http://bittorrent.org/beps/bep_0003.html http://bittorrent.org/beps/bep_0003.html), seem little more than a vague blog post turned in to a "spec". However, somewhat conversely, this has lead to is a wealth of articles describing how to do it. The three guides I used were: - A 2 part blog post which has a bit of a Python bent http://www.kristenwidman.com/blog/33/how-to-write-a-bittorrent-client-part-1/ http://www.kristenwidman.com/blog/33/how-to-write-a-bittorre... - The unofficial specs https://wiki.theory.org/BitTorrentSpecification https://wiki.theory.org/BitTorrentSpecification, and - An incomplete Python client https://github.com/JosephSalisbury/python-bittorrent https://github.com/JosephSalisbury/python-bittorrent I didn't know of the RFC mentioned in the post, that would have also been really useful. A lot of BitTorrent stuff for Python is remarkably hard to find in all the noise of Deluge, the original client, and libtorrent wrappers, but none that existed were sophisticated (or at least well documented) enough for my experiments, they have different focuses. I never went as far as implementing my own BEncoder library, a billion seem to exist in multiple languages and install any BitTorrent Python library and it seems to come with their own copy. (I suspect due to the way BEncoder was bundled in the original client, see: https://pypi.python.org/pypi/bencode https://pypi.python.org/pypi/bencode) I also found a Rust implementation which seems not to compile, but is useful as I'm trying to teach myself Rust https://github.com/kenpratt/rusty_torrent https://github.com/kenpratt/rusty_torrent I think the work to get it to compile might be minimal.
- nickpsecurity 10y ago" this would have been most useful. I'm no C# coder, but it's nicely readable, and I'm sure this is a lot better written up than I could do." I agree. I don't do C# but mostly can follow it. It also is well-organized presentation of much of a protocol all kinds of people keep re-implementing. They need the help more often than not. A great write-up.
- jkeler 10y ago> I also found a Rust implementation which seems not to compile, but is useful as I'm trying to teach myself Rust https://github.com/kenpratt/rusty_torrent https://github.com/kenpratt/rusty_torrent I think the work to get it to compile might be minimal. There is also another project in Rust, it looks more active: https://github.com/GGist/bip-rs https://github.com/GGist/bip-rs It is a collection of libraries. > If anyone who hasn't tried doing this before, the "official" BitTorrent spec docs, namely BEP-3 (http://bittorrent.org/beps/bep_0003.html http://bittorrent.org/beps/bep_0003.html), seem little more than a vague blog post turned in to a "spec". Doesn't look vague at all. What do you think is missing from it?
- nbarbettini 10y agoAwesome write-up! I love C# and this was really well-written. Great work. Pro tip: Use DateTimeOffset instead of DateTime. It's less frustratingly ambiguous than DateTime, and already has a Unix timestamp helper function if you're on the latest framework: https://msdn.microsoft.com/en-us/library/system.datetimeoffset.tounixtimeseconds(v=vs.110).aspx https://msdn.microsoft.com/en-us/library/system.datetimeoffs...
- cheatdeath 10y agoOh cool, thanks! I wasn't aware of DateTimeOffset.
- blt 10y agoIt's really nice to see a walkthrough of a non-trivial program all on one page like this. The clarity of the code and writing makes me want to port it to a different language because it seems like it would be easy with all the needed info in one place.
- th0ma5 10y agoHeh, more than a decade ago I created a torrent file format library in .NET ... actually with VB.Net ... anyway, this is GPLv2 licensed. http://writtorrent.cvs.sourceforge.net/viewvc/writtorrent/writtorrent/torrentclass/torrent.vb?revision=1.1&view=markup http://writtorrent.cvs.sourceforge.net/viewvc/writtorrent/wr...
- ambicapter 10y agoI don't understand this encoding method. If say, a dictionary starts with d and ends with e, how do you know with "d3:key5:valuee" if the value is "value" or "valu"?
- sigcode 10y agohttp://cr.yp.to/proto/netstrings.txt http://cr.yp.to/proto/netstrings.txt (1997)
- nemaar 10y agoThe length of the string is included, so you know where it ends.
- robteix 10y agoBecause the value is actually "5:value", which means it's 5 bytes long. The final 'e' indicates the end of the dictionary, not the value.
- hackeradam17 10y agoI've been considering trying my hand at creating a bittorrent client. This should prove to be most helpful!
- jabstack 10y agoFormer admin of DC# here (forgive the sourceforge hosting-- it was a long time ago! https://sourceforge.net/projects/dc-sharp/ https://sourceforge.net/projects/dc-sharp/). Great write-up! This was a fascinating read- thank you for putting it together. One issue to be mindful of- the HttpWebRequest.BeginGetResponse method does not honor timeouts, and you are on your own to timeout the attempt. Consider using HttpClient, if available in Mono / .NET Core. Otherwise, see MSDN for how to do this: "In the case of asynchronous requests, it is the responsibility of the client application to implement its own time-out mechanism. The following code example shows how to do it." See: https://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetresponse(v=vs.110).aspx https://msdn.microsoft.com/en-us/library/system.net.httpwebr... I'm not sure if you have access to the ThreadPool class. In a bug that Microsoft's library had, I used the TPL Task construct to resolve this. See the pull request here: https://github.com/Microsoft/ProjectOxford-ClientSDK/pull/83/commits/f19b86b74a4841618d7644bcdc249f4c5406d632 https://github.com/Microsoft/ProjectOxford-ClientSDK/pull/83...
- cheatdeath 10y agoThanks for the pointer!
- Const-me 10y agoVery good! A few minor performance comments. 1. In your EncodeDictionary, you sort byte arrays by converting them to string. Correct but subeffective. See e.g. this: http://stackoverflow.com/q/19695629/126995 http://stackoverflow.com/q/19695629/126995 but add checks for nulls, authors of that code forgot about that. 2. You don’t need a dedicated thread to wake up every 1-10 seconds and do something small. Thread are expensive system resources, they own stack, cache misses are guaranteed then they wake up, etc. If your compiler supports async-await, use that instead + endless loop + Task.Delay inside the loop. If not, System.Timers.Timer class will do.
- cheatdeath 10y agoThanks for the tips! I will look into making these modifications when I get a chance.
- Const-me 10y agoYou're welcome. Edited for clarity.
- NKCSS 10y agoFun read, but using automatic properties might lead you down a path that isn't optimal; Take this for example: public byte[] Infohash { get; private set; } = new byte[20]; public string HexStringInfohash { get { return String.Join("", this.Infohash.Select(x => x.ToString("x2"))); } } public string UrlSafeStringInfohash { get { return Encoding.UTF8.GetString(WebUtility.UrlEncodeToBytes(this.Infohash, 0, 20)); } } You have an automatic property and two 'properties' that actually perform work every time you call the getter (might be smarter to make functions of those, so you know it's not just retrieval of data, but work is done). If you were to rewrite this a bit, you could make sure the 'work' is done only when needed, and the properties become actual simple data retrieval properties like: public class Hashes { byte[] _infohash; string _hexStringInfohash, _urlSafeStringInfohash; public byte[] Infohash { get { return _infohash; } private set { _infohash = value; _hexStringInfohash = String.Join("", this.Infohash.Select(x => x.ToString("x2"))); _urlSafeStringInfohash = Encoding.UTF8.GetString(WebUtility.UrlEncodeToBytes(this.Infohash, 0, 20)); } } public string HexStringInfohash { get { return _hexStringInfohash; } } public string UrlSafeStringInfohash { get { return _urlSafeStringInfohash; } } public Hashes() { Infohash = new byte[20]; } } Going further through the article, I spot many more items to improve; but let's not forget your did great work and the code is quite readable. One thing that might help; is building some indexes to know how files are fragmented; you have the following code multiple times: if ((start < Files[i].Offset && end < Files[i].Offset) || (start > Files[i].Offset + Files[i].Size && end > Files[i].Offset + Files[i].Size)) continue; If you'd build an index to know which piece hits which files, you don't have to enumerate this every time. Another general remark is to always 'retrieve' an indexed item from the array and use that instead of keep calling the 'indexed' record. So; do: var file = Files[i]; if ((start < file.Offset && end < file.Offset) || (start > file.Offset + file.Size && end > file.Offset + file.Size)) continue; The code becomes more readable and allows you to change the structure later on more easily since you don't have 100 references tot he same array now and only use an itermediate.
- cheatdeath 10y agoThanks for the comments! I'm all for improved readability. I definitely wasn't aiming for much performance wise, at least initially. These were two areas (of many) that I felt could do with some improvement (especially the file IO). I will look into making some modifications like those suggested when I get a chance.