4 ms·

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - IIRC some clients measure average packet latency and if it starts going up, put some backpressure), resolve magnet URLs, set up port forwarding, reconfigure firewalls, offer API for *arr stack, be a good netizen (report stats correctly, send packets within the specs, do not spam - otherwise other clients will blocklist your in their code or config), implement many BEPs (mutable torrents are cool), be able to recover from interrupted state based on only the data that's on disk, have configurable downloading order (people want to start playing videos before they finish downloading, so you may want to e.g. download header and footer of each file first), and ideally detect duplicates between torrents (cross-seeding). And then there will be people throwing 2TB+ torrents at it (e.g. TLMC) to benchmark it, and saying your client is "literally unusable" if it doesn't handle it. So, building a "perfect" torrent client from the ground up is a daunting task. But the "good" news is that nobody built such a "perfect" client just yet, so if you have some spare months of your time, you can take a shot at it. Or even better yet, open the issue tracker for one of popular clients or libraries, and add one of the missing features from the list above.

thrdbndndn 2y ago
For me, the hardest part of building a good BT client (or any software, really) is getting the UX right. These days, I use QB after uTorrent's downfall, but even after all these years, its UX still isn’t quite there.
- pzmarzly 2y ago
  qBittorrent + VueTorrent is not perfect, but it's hard to find anything better. If you haven't tried it, take a look. https://github.com/VueTorrent/VueTorrent https://github.com/VueTorrent/VueTorrent
  - 6510 2y ago
    https://tixati.com https://tixati.com
    - kohbo 2y ago
      Tixati is simple in the same way Deluge is simple. Because of this they both lack a lot of the quality of life things out of the box that qBittorrent provides.
- szvsw 2y ago
  As someone who stopped torrenting around 2015 or maybe 2016, what happened to uTorrent? Also RIP what.cd, I still am sad that it is gone.
  - kohbo 2y ago
    Became adware
Eikon 2y ago
> but very hard to make a good torrent client > So, building a "perfect" torrent client from the ground up is a daunting task. Don’t you think you’re exaggerating a bit? It’s not daunting by any stretch. The feature set you described is fairly straightforward and something even a beginner developer could tackle without too much hassle. I’m honestly kind of tired of seeing people act like doing anything these days is some impossible feat.
- anacrolix 2y ago
  Amen.
- dgroshev 2y ago
  It's not impossible, it's just hard to be better than libtorrent, otherwise a new client only makes sense as an exercise. Arvid Norberg is hardly a "beginner developer", yet look how hard it is to make a performant implementation of uTP: https://github.com/arvidn/libtorrent/issues/3542 https://github.com/arvidn/libtorrent/issues/3542
- pdimitar 2y ago
  Why would it be an exaggerated claim? Try it yourself, from scratch, and see how quickly you will finish something that will rival and surpass, say, qBittorrent. None of the tasks is breaking new ground or is creatively difficult. But there's a metric ton of those tasks in order to make a truly good torrent client.
bool3max 2y ago
> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?
- andai 2y ago
  Personally, "being as small as uTorrent used to be", but clearly that's not a deal-breaker. (Then again, neither is like two thirds of the stuff GP mentioned!)
  - pzmarzly 2y ago
    To be fair, circa 2010 all torrent clients got "good enough", so my wishlist above is just a wishlist (because torrents are cool and I like to see progress). I use (and sponsor) qBittorrent myself and have no plans to change.
- pzmarzly 2y ago
  I am using qBittorrent, and from the top of my head: - WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision), - cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own) - when it comes to torrent management, there is no way to group torrents into groups with common settings (important if you use multiple private trackers) - people recommend having multiple installations of qBittorrent side by side - when it comes to reconfiguring NAT and firewall, qBittorrent supports UPNP IGD protoocl, but I am not sure about NAT-PMP and PCP - I have never seen qBittorrent connect to a single IPv6 peer - so I don't know if the support is there - download order - you can choose "download in order" or "download rarest first". I dont think "download in order" downloads footers, so mp4 files won't work (IIRC mp4 store metadata in footer, mkv in header)
  - anacrolix 2y ago
    See my other comments, but my client fully supports Webtorrent and webtrackers. Not much support for port forwarding. IPv6 definitely works. Download order is handled by providing readers directly into torrent data and using that for prioritization. So basically request what you need when you need it. No arbitrary list of algorithms.
troupo 2y ago
You may also want both a CLI and a web access/management interface
wslh 2y ago
Yes, even qBitorrent "destroys" your other connection at maximum speeds.
6510 2y ago
Alternatively, a minimal client with few features that can seed large numbers of torrents without crashing or freezing. Settings for how many it may seed at a time, prioritize that what needs seeding, ignore overly seeded things. Options to seed only blocks no one has. Remotely controlled.
- anacrolix 2y ago
  I had a super seeder implementation that seeded over 400k torrents simultaneously using some fancy callbacks at certain points in the protocol. If this is actually a common use case, I can resurrect it into a usable form for the public if there's interest or funding. https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent
  - 6510 2y ago
    https://www.reddit.com/r/DataHoarder/comments/wcusy1/torrent_client_to_support_large_numbers_of/ https://www.reddit.com/r/DataHoarder/comments/wcusy1/torrent... The datahoarders run multiple clients. It's a workable solution but far from ideal.
newsclues 2y ago
Given the state of software quality, it is remarkable that torrent clients are SO GOOD, given they are free!
- anacrolix 2y ago
  There's only a handful of actually complete ones. A few it's surprising they even work, they're crazy complicated, and weird, but once they're stable they need very little maintenance. They're all also written in very different styles due to a mix of C/C++ and very high concurrency.
anacrolix 2y ago
My client anacrolix/torrent has most of that, including streaming, hybrid, v2, all trackers and protocols. I don't do much port forwarding, it's too fiddly. I think I provide upnp out of the box and few other variants that were contributed. I also have all the DHT extensions, like mutable torrents and get/put etc. Large torrents do pop up occasionally, but it's been a long time since someone found a performance issue there but it used to happen every few months. Lots of optimisation. GitHub.com/anacrolix/torrent
- pzmarzly 2y ago
  I know your library does a lot, I have been your GH Sponsor for months ;) Thanks for reminding me about it, I wanted to up the amount. And to thank you for working on such cool stuff and making the ecosystem better. I missed the upnp support, thanks for correcting my beliefs. Which client with WebUI would you recommend the most? I tried using exatorrent and distribyted, but was running into bugs.
- a-french-anon 2y ago
  Any clean room impl. is good news to me, but is the support for private torrent/tracker complete (https://github.com/anacrolix/torrent/issues/531 https://github.com/anacrolix/torrent/issues/531)? NB: especially after https://github.com/Luminarys/synapse https://github.com/Luminarys/synapse died and Transmission "botched" its 4.x rewrite (currently on 3 with some backported patches).
  - transmission3 2y ago
    another person still using transmission 3 here! no clue what i'm going to do if/when it stops compiling.
    - a-french-anon 2y ago
      https://github.com/stefantalpalaru/transmission-og https://github.com/stefantalpalaru/transmission-og
blacklion 2y ago
Also, good disk I/O scheduler needed, or you can thrash even stripped NVMe disks, not to mention spinning rust. Good multi threading without any global locks is a must or your UI/remote control become unresponsive on big number of active torrents. Also, it needs daemon mode, which can be compiled & installed without any GUI libs (like QT, GTK, Waynald, libx11, anything like this) and fast responsive remote GUI for all platforms which allows to manage thousands of torrents simultaneously (I'm looking at you, <any-existing-torrent-client>). It could be Web UI, but for now I didn't se usable one. Only thing that semi-works is Transmission + Transmission-Remote-GUI written in Pascal. But it is effectively abadonware and works worse and worse with each Transmission update. And be scriptable to allow automate change of torrent on tracker, for example (you need to detect it by "Torrent not registered" tracker-specific error answer and re-download torrent building URL from, for example, "comment" field of torrent, again in tracker-specific way).
dgroshev 2y ago
Few more challenges: - seeding a million active torrents (this is entirely possible and needed but drives a lot of architecture) - a UI for a million active torrents - be nice to trackers (keep-alive, batched reports; again, architecture gets tricky, iirc libtorrent can't do that still) - correct pipelining - congestion control on uTP - I/O scheduling/batching - testing all of the above It is quite a lot.
- blacklion 2y ago
  Looks like we have same experience with torrents :-)
  - dgroshev 2y ago
    Yeah. I was working on [0] for a while (I might get back to it when I have more time) and implemented a chunk of the protocol with an eye for a more scalable libtorrent replacement, and it's not entirely trivial. Even things like the piece picker get trickier at scale. [0]: https://dgroshev.com/blog/torrentdyne-encryption/ https://dgroshev.com/blog/torrentdyne-encryption/
vjerancrnjak 2y ago
All of that without encryption. What a meh But shouldn’t be hard to do with modern facilities (async runtimes, streaming libraries etc.) I don’t enjoy many clients yet I use them. It’s definitely a good problem with which to test a programming language. Many fail on parsing, or inefficient network interfaces, inefficient file interfaces, sha sum unusable without intrinsics, etc.

thrdbndndn 2y ago

For me, the hardest part of building a good BT client (or any software, really) is getting the UX right. These days, I use QB after uTorrent's downfall, but even after all these years, its UX still isn’t quite there.

pzmarzly 2y ago
qBittorrent + VueTorrent is not perfect, but it's hard to find anything better. If you haven't tried it, take a look. https://github.com/VueTorrent/VueTorrent https://github.com/VueTorrent/VueTorrent
- 6510 2y ago
  https://tixati.com https://tixati.com
  - kohbo 2y ago
    Tixati is simple in the same way Deluge is simple. Because of this they both lack a lot of the quality of life things out of the box that qBittorrent provides.
szvsw 2y ago
As someone who stopped torrenting around 2015 or maybe 2016, what happened to uTorrent? Also RIP what.cd, I still am sad that it is gone.
- kohbo 2y ago
  Became adware

Eikon 2y ago

> but very hard to make a good torrent client > So, building a "perfect" torrent client from the ground up is a daunting task. Don’t you think you’re exaggerating a bit? It’s not daunting by any stretch. The feature set you described is fairly straightforward and something even a beginner developer could tackle without too much hassle. I’m honestly kind of tired of seeing people act like doing anything these days is some impossible feat.

anacrolix 2y ago
Amen.
dgroshev 2y ago
It's not impossible, it's just hard to be better than libtorrent, otherwise a new client only makes sense as an exercise. Arvid Norberg is hardly a "beginner developer", yet look how hard it is to make a performant implementation of uTP: https://github.com/arvidn/libtorrent/issues/3542 https://github.com/arvidn/libtorrent/issues/3542
pdimitar 2y ago
Why would it be an exaggerated claim? Try it yourself, from scratch, and see how quickly you will finish something that will rival and surpass, say, qBittorrent. None of the tasks is breaking new ground or is creatively difficult. But there's a metric ton of those tasks in order to make a truly good torrent client.

bool3max 2y ago

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

andai 2y ago
Personally, "being as small as uTorrent used to be", but clearly that's not a deal-breaker. (Then again, neither is like two thirds of the stuff GP mentioned!)
- pzmarzly 2y ago
  To be fair, circa 2010 all torrent clients got "good enough", so my wishlist above is just a wishlist (because torrents are cool and I like to see progress). I use (and sponsor) qBittorrent myself and have no plans to change.
pzmarzly 2y ago
I am using qBittorrent, and from the top of my head: - WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision), - cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own) - when it comes to torrent management, there is no way to group torrents into groups with common settings (important if you use multiple private trackers) - people recommend having multiple installations of qBittorrent side by side - when it comes to reconfiguring NAT and firewall, qBittorrent supports UPNP IGD protoocl, but I am not sure about NAT-PMP and PCP - I have never seen qBittorrent connect to a single IPv6 peer - so I don't know if the support is there - download order - you can choose "download in order" or "download rarest first". I dont think "download in order" downloads footers, so mp4 files won't work (IIRC mp4 store metadata in footer, mkv in header)
- anacrolix 2y ago
  See my other comments, but my client fully supports Webtorrent and webtrackers. Not much support for port forwarding. IPv6 definitely works. Download order is handled by providing readers directly into torrent data and using that for prioritization. So basically request what you need when you need it. No arbitrary list of algorithms.

troupo 2y ago

You may also want both a CLI and a web access/management interface

wslh 2y ago

Yes, even qBitorrent "destroys" your other connection at maximum speeds.

6510 2y ago

Alternatively, a minimal client with few features that can seed large numbers of torrents without crashing or freezing. Settings for how many it may seed at a time, prioritize that what needs seeding, ignore overly seeded things. Options to seed only blocks no one has. Remotely controlled.

anacrolix 2y ago
I had a super seeder implementation that seeded over 400k torrents simultaneously using some fancy callbacks at certain points in the protocol. If this is actually a common use case, I can resurrect it into a usable form for the public if there's interest or funding. https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent
- 6510 2y ago
  https://www.reddit.com/r/DataHoarder/comments/wcusy1/torrent_client_to_support_large_numbers_of/ https://www.reddit.com/r/DataHoarder/comments/wcusy1/torrent... The datahoarders run multiple clients. It's a workable solution but far from ideal.

newsclues 2y ago

Given the state of software quality, it is remarkable that torrent clients are SO GOOD, given they are free!

anacrolix 2y ago
There's only a handful of actually complete ones. A few it's surprising they even work, they're crazy complicated, and weird, but once they're stable they need very little maintenance. They're all also written in very different styles due to a mix of C/C++ and very high concurrency.

anacrolix 2y ago

My client anacrolix/torrent has most of that, including streaming, hybrid, v2, all trackers and protocols. I don't do much port forwarding, it's too fiddly. I think I provide upnp out of the box and few other variants that were contributed. I also have all the DHT extensions, like mutable torrents and get/put etc. Large torrents do pop up occasionally, but it's been a long time since someone found a performance issue there but it used to happen every few months. Lots of optimisation. GitHub.com/anacrolix/torrent

pzmarzly 2y ago
I know your library does a lot, I have been your GH Sponsor for months ;) Thanks for reminding me about it, I wanted to up the amount. And to thank you for working on such cool stuff and making the ecosystem better. I missed the upnp support, thanks for correcting my beliefs. Which client with WebUI would you recommend the most? I tried using exatorrent and distribyted, but was running into bugs.
a-french-anon 2y ago
Any clean room impl. is good news to me, but is the support for private torrent/tracker complete (https://github.com/anacrolix/torrent/issues/531 https://github.com/anacrolix/torrent/issues/531)? NB: especially after https://github.com/Luminarys/synapse https://github.com/Luminarys/synapse died and Transmission "botched" its 4.x rewrite (currently on 3 with some backported patches).
- transmission3 2y ago
  another person still using transmission 3 here! no clue what i'm going to do if/when it stops compiling.
  - a-french-anon 2y ago
    https://github.com/stefantalpalaru/transmission-og https://github.com/stefantalpalaru/transmission-og

blacklion 2y ago

Also, good disk I/O scheduler needed, or you can thrash even stripped NVMe disks, not to mention spinning rust. Good multi threading without any global locks is a must or your UI/remote control become unresponsive on big number of active torrents. Also, it needs daemon mode, which can be compiled & installed without any GUI libs (like QT, GTK, Waynald, libx11, anything like this) and fast responsive remote GUI for all platforms which allows to manage thousands of torrents simultaneously (I'm looking at you, <any-existing-torrent-client>). It could be Web UI, but for now I didn't se usable one. Only thing that semi-works is Transmission + Transmission-Remote-GUI written in Pascal. But it is effectively abadonware and works worse and worse with each Transmission update. And be scriptable to allow automate change of torrent on tracker, for example (you need to detect it by "Torrent not registered" tracker-specific error answer and re-download torrent building URL from, for example, "comment" field of torrent, again in tracker-specific way).

dgroshev 2y ago

Few more challenges: - seeding a million active torrents (this is entirely possible and needed but drives a lot of architecture) - a UI for a million active torrents - be nice to trackers (keep-alive, batched reports; again, architecture gets tricky, iirc libtorrent can't do that still) - correct pipelining - congestion control on uTP - I/O scheduling/batching - testing all of the above It is quite a lot.

blacklion 2y ago
Looks like we have same experience with torrents :-)
- dgroshev 2y ago
  Yeah. I was working on [0] for a while (I might get back to it when I have more time) and implemented a chunk of the protocol with an eye for a more scalable libtorrent replacement, and it's not entirely trivial. Even things like the piece picker get trickier at scale. [0]: https://dgroshev.com/blog/torrentdyne-encryption/ https://dgroshev.com/blog/torrentdyne-encryption/

vjerancrnjak 2y ago

All of that without encryption. What a meh But shouldn’t be hard to do with modern facilities (async runtimes, streaming libraries etc.) I don’t enjoy many clients yet I use them. It’s definitely a good problem with which to test a programming language. Many fail on parsing, or inefficient network interfaces, inefficient file interfaces, sha sum unusable without intrinsics, etc.