5 ms·

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a colle

by lqet 2y ago

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often noticed that when I set the maximum upstream to 0, clients would stop sending me pieces. Do clients share a predefined list of configuration parameters with each other or with the tracker (Max upstream, max downstream, etc.)? Or is it something more sophisticated?

Trung0246 2y ago
There's private tracker which forces their user to upload.
ratorx 2y ago
IIRC, when both peers only have some subset of pieces, they are meant to “trade” with each other, and so will eventually drop low value peers for higher value ones. Aside from the adversarial case, this helps optimise for being nearer to the people you are sharing with, since torrents are global. In practice, BitTorrent really needs seeders who have downloaded the entirety of the file to be fast for everything except really popular downloads, Seeders don’t really check for fairness and will typically upload to whoever they can the fastest (with a limit on number of peers). There’s an even more adversarial case because the unit of validation (a piece with a fixed hash from the spec) might be bigger than the chunks that are being shared individually. So it’s possible for a peer to fake having pieces and upload garbage data instead, and they wouldn’t be caught since different chunks came from different peers.
- anacrolix 2y ago
  For the regular case, 5-10 seeders will hold up an entire torrent indefinitely and without any issues. The contribution leechers make drops off very quickly if good seeders exist. My algorithm did favour the best peers (both upload and download) but a few years ago switched to a "seeded" ordering to prevent bad behaviour clients dumping or starving new peers. The garbage uploaders are not an issue. You can isolate peers to pieces and remove them with certainty, or use a technique called smartban which uncovers bad peers very quickly.
  - ratorx 2y ago
    Sure, I didn’t mean to suggest that it was impossible to handle, just that it was an edge case that you needed to explicitly handle and which a minimal implementation of the protocol might be likely to leave out.
- __MatrixMan__ 2y ago
  Wouldn't the peer uploading garbage data be caught immediately because the garbage wouldn't hash to the correct value? I imagine most clients would stop talking to such a peer rather quickly.
  - ratorx 2y ago
    Torrents are split into pieces, each of which has a hash that allows it to be validated independently. However, these pieces can themselves be large, so it is often recommended to split them into smaller chunks that are sent one at a time. You cannot validate a chunk independently, and if you assemble a piece from chunks from multiple origins, you can’t immediately tell which origin is bad. You need to explicitly handle this case, like the sibling comment mentioned by isolating peers who may have sent bad data and forcing validation.
jedimastert 2y ago
> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download). It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced https://en.wikipedia.org/wiki/Leecher_(computing) https://en.wikipedia.org/wiki/Leecher_(computing)
- hoseja 2y ago
  I have the opposite problem, many things I download seem so well-seeded (seedboxes?) that I rarely achieve any sort of respectable ratio even if my uplink idles most of the time. It might be client problem but I haven't found much anyone discussing this. TAKE MY BITS!
  - Akronymus 2y ago
    Over here (austria), you are pretty free to leech as much as you want, but as soon as you actually start seeding at any capacity, people start paying very close attention.
  - anacrolix 2y ago
    See my other comment, good seeders carry most of the network these days. Upload bandwidth has become huge since BitTorrent first debuted so true P2P for resource reasons is not really a necessity anymore.
  - __MatrixMan__ 2y ago
    I've often considered writing the equivalent of a robotrader for this. If you're paying for a sedbox and want to use it to the fullest re: maximizing your ratio then some automated attempt to predict demand and start seeding the right thing early would likely go a long way. Unlike predicting the stock market, competition wouldn't be very fierce, and I think the tracker API would give you most of what you need. It's in the long list of things I'll do if I ever find a big pile of time.
  - rakoo 2y ago
    If the torrent is already well-seeded it's ok, a 1567th seed won't make a difference so don't bother with them. It's the less seeded torrents where you can put some efforts and do something good.
rakoo 2y ago
Sufficiently evolved torrent clients follow a tit-for-tat protocol: - If you're nice with me I'll be nice with you - If you're mean with me I'll be mean with you This is the best answer for the Prisoner's dilemma (https://en.wikipedia.org/wiki/Prisoner's_dilemma https://en.wikipedia.org/wiki/Prisoner's_dilemma) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate. In bittorrent it's typically implemented as follows: peers start by sending a very minimal content, and see what the other replies. If they reply with low enough latency then slowly increase the amount of content that is sent and see if they reply with the same increased amount of content; if they do, continue up to the max of what the network link allows (in combination with other peers of course). If at some point the other peer doesn't send something equivalent (even though we know they have it and we asked for it) then that peer can be cut off for not being cooperative. Situation is different for seeds of course, because they have everything and want nothing, but they can have a similar behaviour -- start sending a little, increase slowly over time
- BonoboIO 2y ago
  Did knew, that the client does that with the tit-for-that. In an ideal world the network would prioritize to upload to high speed like webseeds, so the webseeds can distribute even faster for everyone else. Or do I make a mistake in my mind and the network already distributes efficiently and it is not needed.
  - rakoo 2y ago
    A single node distributing everything "because it's fast" isn't really the architecture that bittorrent is aiming for, and I don't think it's one we should be aiming for. Try downloading any well distributed torrent and you'll see your bandwidth capacity automatically maxxed out. The gradual decision with an emerging selection of better sources always leads to that. Bittorrent doesn't really have an efficiency problem
- andai 2y ago
  > [tit for tat] is the best answer for the Prisoner's dilemma on the long run This is fascinating. Have there been any simulations about this? I'm sure they've looked into this in game theory, but I'm wondering if you have a big swarm of torrent clients, does the scale of the population change the outcome, or is it the same? (It seems like it would be wise when designing the torrent clients to run such simulations, so I'm thinking this probably has been done at some point.)

by lqet 2y ago

Trung0246 2y ago

There's private tracker which forces their user to upload.

ratorx 2y ago

IIRC, when both peers only have some subset of pieces, they are meant to “trade” with each other, and so will eventually drop low value peers for higher value ones. Aside from the adversarial case, this helps optimise for being nearer to the people you are sharing with, since torrents are global. In practice, BitTorrent really needs seeders who have downloaded the entirety of the file to be fast for everything except really popular downloads, Seeders don’t really check for fairness and will typically upload to whoever they can the fastest (with a limit on number of peers). There’s an even more adversarial case because the unit of validation (a piece with a fixed hash from the spec) might be bigger than the chunks that are being shared individually. So it’s possible for a peer to fake having pieces and upload garbage data instead, and they wouldn’t be caught since different chunks came from different peers.

anacrolix 2y ago
For the regular case, 5-10 seeders will hold up an entire torrent indefinitely and without any issues. The contribution leechers make drops off very quickly if good seeders exist. My algorithm did favour the best peers (both upload and download) but a few years ago switched to a "seeded" ordering to prevent bad behaviour clients dumping or starving new peers. The garbage uploaders are not an issue. You can isolate peers to pieces and remove them with certainty, or use a technique called smartban which uncovers bad peers very quickly.
- ratorx 2y ago
  Sure, I didn’t mean to suggest that it was impossible to handle, just that it was an edge case that you needed to explicitly handle and which a minimal implementation of the protocol might be likely to leave out.
__MatrixMan__ 2y ago
Wouldn't the peer uploading garbage data be caught immediately because the garbage wouldn't hash to the correct value? I imagine most clients would stop talking to such a peer rather quickly.
- ratorx 2y ago
  Torrents are split into pieces, each of which has a hash that allows it to be validated independently. However, these pieces can themselves be large, so it is often recommended to split them into smaller chunks that are sent one at a time. You cannot validate a chunk independently, and if you assemble a piece from chunks from multiple origins, you can’t immediately tell which origin is bad. You need to explicitly handle this case, like the sibling comment mentioned by isolating peers who may have sent bad data and forcing validation.

jedimastert 2y ago

> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download). It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced https://en.wikipedia.org/wiki/Leecher_(computing) https://en.wikipedia.org/wiki/Leecher_(computing)

hoseja 2y ago
I have the opposite problem, many things I download seem so well-seeded (seedboxes?) that I rarely achieve any sort of respectable ratio even if my uplink idles most of the time. It might be client problem but I haven't found much anyone discussing this. TAKE MY BITS!
- Akronymus 2y ago
  Over here (austria), you are pretty free to leech as much as you want, but as soon as you actually start seeding at any capacity, people start paying very close attention.
- anacrolix 2y ago
  See my other comment, good seeders carry most of the network these days. Upload bandwidth has become huge since BitTorrent first debuted so true P2P for resource reasons is not really a necessity anymore.
- __MatrixMan__ 2y ago
  I've often considered writing the equivalent of a robotrader for this. If you're paying for a sedbox and want to use it to the fullest re: maximizing your ratio then some automated attempt to predict demand and start seeding the right thing early would likely go a long way. Unlike predicting the stock market, competition wouldn't be very fierce, and I think the tracker API would give you most of what you need. It's in the long list of things I'll do if I ever find a big pile of time.
- rakoo 2y ago
  If the torrent is already well-seeded it's ok, a 1567th seed won't make a difference so don't bother with them. It's the less seeded torrents where you can put some efforts and do something good.

rakoo 2y ago

Sufficiently evolved torrent clients follow a tit-for-tat protocol: - If you're nice with me I'll be nice with you - If you're mean with me I'll be mean with you This is the best answer for the Prisoner's dilemma (https://en.wikipedia.org/wiki/Prisoner's_dilemma https://en.wikipedia.org/wiki/Prisoner's_dilemma) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate. In bittorrent it's typically implemented as follows: peers start by sending a very minimal content, and see what the other replies. If they reply with low enough latency then slowly increase the amount of content that is sent and see if they reply with the same increased amount of content; if they do, continue up to the max of what the network link allows (in combination with other peers of course). If at some point the other peer doesn't send something equivalent (even though we know they have it and we asked for it) then that peer can be cut off for not being cooperative. Situation is different for seeds of course, because they have everything and want nothing, but they can have a similar behaviour -- start sending a little, increase slowly over time

BonoboIO 2y ago
Did knew, that the client does that with the tit-for-that. In an ideal world the network would prioritize to upload to high speed like webseeds, so the webseeds can distribute even faster for everyone else. Or do I make a mistake in my mind and the network already distributes efficiently and it is not needed.
- rakoo 2y ago
  A single node distributing everything "because it's fast" isn't really the architecture that bittorrent is aiming for, and I don't think it's one we should be aiming for. Try downloading any well distributed torrent and you'll see your bandwidth capacity automatically maxxed out. The gradual decision with an emerging selection of better sources always leads to that. Bittorrent doesn't really have an efficiency problem
andai 2y ago
> [tit for tat] is the best answer for the Prisoner's dilemma on the long run This is fascinating. Have there been any simulations about this? I'm sure they've looked into this in game theory, but I'm wondering if you have a big swarm of torrent clients, does the scale of the population change the outcome, or is it the same? (It seems like it would be wise when designing the torrent clients to run such simulations, so I'm thinking this probably has been done at some point.)