5 ms·

Awesome write-up and a great way to dig into the inner-workings of BitTorrent. Nice work! Glancing at the wiki page for BitTorrent[0], I'm a bit surprised that

Awesome write-up and a great way to dig into the inner-workings of BitTorrent. Nice work! Glancing at the wiki page for BitTorrent[0], I'm a bit surprised that there isn't more of an effort to create cross-platform libraries and clients using Go or Rust for this. Seems like a perfect use case? For example, synapse [1]. [0] https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients#Libraries https://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clien... [1] https://github.com/Luminarys/synapse https://github.com/Luminarys/synapse

eat_veggies 7y ago
I think BitTorrent is a very 2000's/early 2010's technology. That's when it feels like it peaked, at least. The future is probably more toward WebTorrents [0] (with WebRTC) and IPFS [1]. And those have great cross platform libraries. [0] https://webtorrent.io/ https://webtorrent.io/ [1] https://ipfs.io/ https://ipfs.io/
- totalperspectiv 7y ago
  As a torrent newb, what's the tldr on on why webtorrent over BitTorrent?
  - eat_veggies 7y ago
    This is totally just my opinion, but the days of the mainstream consumer having a dedicated torrent client installed on their desktop computer (or even as a phone app) are pretty much over. But as a streaming technology, torrents are still pretty great. And like it or not, being able to run in browsers is a major advantage right now.
    - 3fe9a03ccd14ca5 7y ago
      This is true for the streaming use case, where you have a browser open and you want to watch a video. The case the author is going after is downloading a large Debian distro, so in the use-case of large, distributed file distribution there will always be solutions like the author.
      - eat_veggies 7y ago
        How many end users actually download linux distributions? I'd argue that most mainstream consumption is: 1. video streaming 2. centralized sources like news websites 3. sharing in group chats End users don't really download large files like Debian outside of gaming and OS updates. To that end, Blizzard Downloader [0] uses BitTorrent, and Windows 10 uses some similar p2p system [1] [0] https://wow.gamepedia.com/Blizzard_Downloader https://wow.gamepedia.com/Blizzard_Downloader [1] https://www.pcworld.com/article/2955491/how-to-stop-windows-10-from-using-your-pcs-bandwidth-to-update-strangers-systems.html https://www.pcworld.com/article/2955491/how-to-stop-windows-...
        gsich 7y ago
        Not anymore since the Battle.net launcher.
    - Thaxll 7y ago
      Over? With all the new vod services like Netflix, Disney+, Amazon Prime ect ... It all brings back cable souvenirs, meaning than Torrent is stronger than ever.
      - phnofive 7y ago
        Probably not what you meant, but I’d think a licensing and technology arrangement that allowed VoD services to deliver content this way (like Skype when it was good) and use customers as a second line CDN would be awesome.
        catalogia 7y ago
        If customers would be subsidizing the corporation's bandwidth costs, I hope they'd be getting a good discount over what a centralized distribution service with the same licensed content would cost. Donating bandwidth to a community torrent for the general good is not really the same thing as donating bandwidth to a for-profit corporation. I'm much more inclined to do the former than the later.
        phnofive 7y ago
        This would ultimately be in the form of not further raising prices, and become a standard thing. Don’t have >15 Mbps up? No problem, extra $3 per month.
        Moru 7y ago
        Didn't some mmorpgs do this to distribute updates over the client while you were playing?
        mebo 7y ago
        Back in the 2010s, yes, not anymore though afaik, I assume that CDNs/traffic got cheap enough to not go that far. Microsoft actually introduced P2P update sharing with Windows Update for Windows 10, I think it's enabled by default.
  - giancarlostoro 7y ago
    WebTorrent should just work on a browser, no special client necessary. This will allow projects like PeerTube to create websites where the primary content is decentralized by viewers kind of thing.
    - bl4ckm0r3 7y ago
      instant.io and webtor.io are two great examples of in-browser webtorrent streaming/dowmload sites!
    - lossolo 7y ago
      This will be hard to do relying only on WebTorrent browser based client, because if no one is watching particular video, there will be no seeds. You close browser = close WebTorrent = you stop seeding You stop watching video = you stop seeding it Long and large video file? = if most people watch at half of video, then data from beginning is already out of the buffer = no seed for first half of video. To this day there is still no support for WebTorrent in other popular Bittorrent clients so they can't connect to each other. But there is some movement in libtorrent library so we will see.
      - ficklepickle 7y ago
        That makes sense. I was wondering how browsers were communicating with UDP peers... Turns out they aren't! A quick glance at the webtorrent readme says it uses UDP on node. There is a webtorrent desktop app in beta that does both.[0] Here's hoping it makes it to libtorrent. Even then, I'd imagine it would take quite a while to make it into the clients running on seedboxes, desktops, etc.
    - jekrb 7y ago
      Brave Browser has WebTorrent built in. Just paste a magnet link into the URL bar and it will start downloading/seeding.
      - ignoramous 7y ago
        Alternatively, one could use https://instant.io https://instant.io from any browser to seed or download. Makes for a good alternative to https://send.firefox.com https://send.firefox.com for really large files.
  - gpm 7y ago
    Why would I download a sketchy ass torrent client when I can go to a website, download the torrent in my browser, and be able to rely on my browsers sandboxing against the torrent client? Even better, if it's video I'm torrenting, I can rely on my browsers sandboxing against the torrents contents too. It's more convenient, it's safer, it sucks for other torrentors because I don't seed as long, but let's be honest, most people downloading torrents don't care.
    - G4E 7y ago
      Why a torrent client would be sketchier than any other software ? There are plethora of good open source clients like qbittorrent, transmission ... even aria2 ! At that rate, are you suspicious of wget too ? Even more of Chrome ? There are a lot of sketchy things going on in that one ! It is the user's responsability to choose in which software he trust, and a bittorent client is not worst than anything else ?
      - xvector 7y ago
        Browsers are far more battle-tested than just about any other web-facing application on your computer. Of course, you could make the personal decision to trust a client, and that is fine. But if you aren’t willing to blindly trust a client, the other guy’s point still stands - browsers are probably just the better choice here from a security POV.
        slyall 7y ago
        But a bittorrent client isn't trying to be a browser and all the complex stuff that requires. All it's doing is downloading bittorrent files and having a usable GUI. And as the original article demonstrated the first half of that is a weekend project.
        pdkl95 7y ago
        > Browsers are far more battle-tested than just about any other web-facing application on your computer. They also have a monstrous attack surface because they are "web-facing". A specialized client that only implements one protocol without any connection to the "web" is far easier to reason about and debug. If you only consider the number of man-years an application has been battle-tested, you imply that design complexity and attack surface doesn't matter. If we account for complexity by using a metric like "(man-years of battle-testing)/(magnitude of attack surface)", a well-tested specialized client that hasn't had many recent bug reports is a much safer choice than anything running in a browser. > blindly trust a client That's even worse for the browser: you have to trust several orders of magnitude more code implementing a massive set of interdependent features. Yes, there are probably a lot more people working on fixing bugs in the browser, but there are also a lot of people adding/modifying features and thus creating new bugs.
  - swiley 7y ago
    iPhone.
  - bitwize 7y ago
    It's the Web. Protocols other than HTTP are old hat, dude.
  - conradev 7y ago
    WebTorrent is a version of the BitTorrent protocol modified to work over WebRTC so that it can work in a browser. Someone is actively working with the maintainer of libtorrent (popular implementation of BitTorrent) to get support for it into the library: https://github.com/arvidn/libtorrent/pull/4123 https://github.com/arvidn/libtorrent/pull/4123
    - dastx 7y ago
      Wow that's a whole lot of progress in very little time. I've been waiting for this crossover since the early days of WebTorrent. Once enough popular clients implement this, in theory all clients could simply move over to using web sockets and deprecate the old methods.
      - ignoramous 7y ago
        Genuinely curious: Why are the old methods [0] not upto the mark? What extra do WebSockets [1] bring to the table? [0] What are those? Plain Old TCP Sockets? [1] You mean, WebRTC, right?
        noisem4ker 7y ago
        WebRTC works in browsers. This alone is enough to trump every other consideration these days.
        dastx 7y ago
        It's not that they aren't good enough. It uses both TCP and UDP (and μTP if enabled). Perfectly fine for it's use case. I guess my thinking is more along the lines of, if everyone is using WebRTC (yes you're right, WebRTC not WebSockets, my mistake), there is no need to use and maintain multiple channels. If WebRTC works well enough, over time, once everyone is able to use WebRTC, you can trim a lot of the TCP code and thus keep maintenance overheads much lower. Especially when more peers can be included in your pool. This would obviously require majority of the torrent clients to implement WebTorrents as otherwise it wouldn't make sense and you'd exclude more than you would include. If I'm not mistaken, libtorrent is used among many of the open source torrent clients, and so I think merging this code would suddenly give WebTorrent users an enormous amount of peers once they all upgrade. For client implementations upgrading is probably requires next to no code changes.
- tryptophan 7y ago
  Isn't webtorrent just bittorrent implemented in JS so it can run on browsers?
  - slimsag 7y ago
    Yes, it is.
  - the8472 7y ago
    webtorrent is not compatible with bittorrent because browsers can't speak raw TCP/UDP as required by the bittorrent protocol. They use webrtc, which is not compatible with existing bittorrent clients and requires some centralized components (signalling servers) while bittorrent can operate fully decentralized. Also running things in a browser is at cross purposes with torrent long-term availability, large-scale (giga or terabyte) file management, efficient IO and so on.
lossolo 7y ago
This is mature Go bittorrent library: https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent
- devadvance 7y ago
  I didn't realize this existed, thanks for the link! It'll be a great codebase to look through :)
  - trevyn 7y ago
    See also: https://github.com/cenkalti/rain https://github.com/cenkalti/rain

eat_veggies 7y ago

I think BitTorrent is a very 2000's/early 2010's technology. That's when it feels like it peaked, at least. The future is probably more toward WebTorrents [0] (with WebRTC) and IPFS [1]. And those have great cross platform libraries. [0] https://webtorrent.io/ https://webtorrent.io/ [1] https://ipfs.io/ https://ipfs.io/

totalperspectiv 7y ago
As a torrent newb, what's the tldr on on why webtorrent over BitTorrent?
- eat_veggies 7y ago
  This is totally just my opinion, but the days of the mainstream consumer having a dedicated torrent client installed on their desktop computer (or even as a phone app) are pretty much over. But as a streaming technology, torrents are still pretty great. And like it or not, being able to run in browsers is a major advantage right now.
  - 3fe9a03ccd14ca5 7y ago
    This is true for the streaming use case, where you have a browser open and you want to watch a video. The case the author is going after is downloading a large Debian distro, so in the use-case of large, distributed file distribution there will always be solutions like the author.
    - eat_veggies 7y ago
      How many end users actually download linux distributions? I'd argue that most mainstream consumption is: 1. video streaming 2. centralized sources like news websites 3. sharing in group chats End users don't really download large files like Debian outside of gaming and OS updates. To that end, Blizzard Downloader [0] uses BitTorrent, and Windows 10 uses some similar p2p system [1] [0] https://wow.gamepedia.com/Blizzard_Downloader https://wow.gamepedia.com/Blizzard_Downloader [1] https://www.pcworld.com/article/2955491/how-to-stop-windows-10-from-using-your-pcs-bandwidth-to-update-strangers-systems.html https://www.pcworld.com/article/2955491/how-to-stop-windows-...
      - gsich 7y ago
        Not anymore since the Battle.net launcher.
  - Thaxll 7y ago
    Over? With all the new vod services like Netflix, Disney+, Amazon Prime ect ... It all brings back cable souvenirs, meaning than Torrent is stronger than ever.
    - phnofive 7y ago
      Probably not what you meant, but I’d think a licensing and technology arrangement that allowed VoD services to deliver content this way (like Skype when it was good) and use customers as a second line CDN would be awesome.
      - catalogia 7y ago
        If customers would be subsidizing the corporation's bandwidth costs, I hope they'd be getting a good discount over what a centralized distribution service with the same licensed content would cost. Donating bandwidth to a community torrent for the general good is not really the same thing as donating bandwidth to a for-profit corporation. I'm much more inclined to do the former than the later.
        phnofive 7y ago
        This would ultimately be in the form of not further raising prices, and become a standard thing. Don’t have >15 Mbps up? No problem, extra $3 per month.
      - Moru 7y ago
        Didn't some mmorpgs do this to distribute updates over the client while you were playing?
        mebo 7y ago
        Back in the 2010s, yes, not anymore though afaik, I assume that CDNs/traffic got cheap enough to not go that far. Microsoft actually introduced P2P update sharing with Windows Update for Windows 10, I think it's enabled by default.
- giancarlostoro 7y ago
  WebTorrent should just work on a browser, no special client necessary. This will allow projects like PeerTube to create websites where the primary content is decentralized by viewers kind of thing.
  - bl4ckm0r3 7y ago
    instant.io and webtor.io are two great examples of in-browser webtorrent streaming/dowmload sites!
  - lossolo 7y ago
    This will be hard to do relying only on WebTorrent browser based client, because if no one is watching particular video, there will be no seeds. You close browser = close WebTorrent = you stop seeding You stop watching video = you stop seeding it Long and large video file? = if most people watch at half of video, then data from beginning is already out of the buffer = no seed for first half of video. To this day there is still no support for WebTorrent in other popular Bittorrent clients so they can't connect to each other. But there is some movement in libtorrent library so we will see.
    - ficklepickle 7y ago
      That makes sense. I was wondering how browsers were communicating with UDP peers... Turns out they aren't! A quick glance at the webtorrent readme says it uses UDP on node. There is a webtorrent desktop app in beta that does both.[0] Here's hoping it makes it to libtorrent. Even then, I'd imagine it would take quite a while to make it into the clients running on seedboxes, desktops, etc.
  - jekrb 7y ago
    Brave Browser has WebTorrent built in. Just paste a magnet link into the URL bar and it will start downloading/seeding.
    - ignoramous 7y ago
      Alternatively, one could use https://instant.io https://instant.io from any browser to seed or download. Makes for a good alternative to https://send.firefox.com https://send.firefox.com for really large files.
- gpm 7y ago
  Why would I download a sketchy ass torrent client when I can go to a website, download the torrent in my browser, and be able to rely on my browsers sandboxing against the torrent client? Even better, if it's video I'm torrenting, I can rely on my browsers sandboxing against the torrents contents too. It's more convenient, it's safer, it sucks for other torrentors because I don't seed as long, but let's be honest, most people downloading torrents don't care.
  - G4E 7y ago
    Why a torrent client would be sketchier than any other software ? There are plethora of good open source clients like qbittorrent, transmission ... even aria2 ! At that rate, are you suspicious of wget too ? Even more of Chrome ? There are a lot of sketchy things going on in that one ! It is the user's responsability to choose in which software he trust, and a bittorent client is not worst than anything else ?
    - xvector 7y ago
      Browsers are far more battle-tested than just about any other web-facing application on your computer. Of course, you could make the personal decision to trust a client, and that is fine. But if you aren’t willing to blindly trust a client, the other guy’s point still stands - browsers are probably just the better choice here from a security POV.
      - slyall 7y ago
        But a bittorrent client isn't trying to be a browser and all the complex stuff that requires. All it's doing is downloading bittorrent files and having a usable GUI. And as the original article demonstrated the first half of that is a weekend project.
      - pdkl95 7y ago
        > Browsers are far more battle-tested than just about any other web-facing application on your computer. They also have a monstrous attack surface because they are "web-facing". A specialized client that only implements one protocol without any connection to the "web" is far easier to reason about and debug. If you only consider the number of man-years an application has been battle-tested, you imply that design complexity and attack surface doesn't matter. If we account for complexity by using a metric like "(man-years of battle-testing)/(magnitude of attack surface)", a well-tested specialized client that hasn't had many recent bug reports is a much safer choice than anything running in a browser. > blindly trust a client That's even worse for the browser: you have to trust several orders of magnitude more code implementing a massive set of interdependent features. Yes, there are probably a lot more people working on fixing bugs in the browser, but there are also a lot of people adding/modifying features and thus creating new bugs.
- swiley 7y ago
  iPhone.
- bitwize 7y ago
  It's the Web. Protocols other than HTTP are old hat, dude.
- conradev 7y ago
  WebTorrent is a version of the BitTorrent protocol modified to work over WebRTC so that it can work in a browser. Someone is actively working with the maintainer of libtorrent (popular implementation of BitTorrent) to get support for it into the library: https://github.com/arvidn/libtorrent/pull/4123 https://github.com/arvidn/libtorrent/pull/4123
  - dastx 7y ago
    Wow that's a whole lot of progress in very little time. I've been waiting for this crossover since the early days of WebTorrent. Once enough popular clients implement this, in theory all clients could simply move over to using web sockets and deprecate the old methods.
    - ignoramous 7y ago
      Genuinely curious: Why are the old methods [0] not upto the mark? What extra do WebSockets [1] bring to the table? [0] What are those? Plain Old TCP Sockets? [1] You mean, WebRTC, right?
      - noisem4ker 7y ago
        WebRTC works in browsers. This alone is enough to trump every other consideration these days.
      - dastx 7y ago
        It's not that they aren't good enough. It uses both TCP and UDP (and μTP if enabled). Perfectly fine for it's use case. I guess my thinking is more along the lines of, if everyone is using WebRTC (yes you're right, WebRTC not WebSockets, my mistake), there is no need to use and maintain multiple channels. If WebRTC works well enough, over time, once everyone is able to use WebRTC, you can trim a lot of the TCP code and thus keep maintenance overheads much lower. Especially when more peers can be included in your pool. This would obviously require majority of the torrent clients to implement WebTorrents as otherwise it wouldn't make sense and you'd exclude more than you would include. If I'm not mistaken, libtorrent is used among many of the open source torrent clients, and so I think merging this code would suddenly give WebTorrent users an enormous amount of peers once they all upgrade. For client implementations upgrading is probably requires next to no code changes.
tryptophan 7y ago
Isn't webtorrent just bittorrent implemented in JS so it can run on browsers?
- slimsag 7y ago
  Yes, it is.
- the8472 7y ago
  webtorrent is not compatible with bittorrent because browsers can't speak raw TCP/UDP as required by the bittorrent protocol. They use webrtc, which is not compatible with existing bittorrent clients and requires some centralized components (signalling servers) while bittorrent can operate fully decentralized. Also running things in a browser is at cross purposes with torrent long-term availability, large-scale (giga or terabyte) file management, efficient IO and so on.

lossolo 7y ago

This is mature Go bittorrent library: https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent

devadvance 7y ago
I didn't realize this existed, thanks for the link! It'll be a great codebase to look through :)
- trevyn 7y ago
  See also: https://github.com/cenkalti/rain https://github.com/cenkalti/rain