9 ms·

I'm deeply concerned about popularizing torrent streaming. A few leechers breaking "rarest-first" might not cause much harm. But if most leechers become stream

by MoSal 10y ago

I'm deeply concerned about popularizing torrent streaming. A few leechers breaking "rarest-first" might not cause much harm. But if most leechers become streamers, torrents will lose their efficiency in distributing less-popular content. Transmission implemented a "streaming" feature once, that didn't actually stream. It just stopped fetching pieces "with the same rarity" randomly. They still got too much heat for adding that not very usefull feature. And they reverted the commits soon after.

catwell 10y ago
This could be true, however I think that if all this torrent streaming sticks to same behavior such as that of XBMC torrent [1] (i.e. streaming for the whole duration of the movie) the swarm will end up stronger, not weaker. [1] https://github.com/steeve/xbmctorrent/blob/e6bcb1037668959e1e3cb5ba8cf3e379c6638da9/README.md#doesnt-sequential-download-on-bittorrent-is-bad https://github.com/steeve/xbmctorrent/blob/e6bcb1037668959e1...
- MoSal 10y ago
  Suppose there is only one seeder and 2-3 leechers. All leechers managed to get 50+%. Now, if the seeder disappears, the leechers should be able to finish the download from each other. If all initial leechers are streamers, that download will never finish for them or anyone else. Supposedly long seed intervals will not help the swarm in that case.
  - RyanZAG 10y ago
    Streaming isn't going to work there anyway. In reality, there are 100+ seeders and if you had a constant supply of streamers who spend half their time seeding (assuming they download the whole movie by the 50% mark), then you have a very very healthy swarm.
deepnet 10y ago
Unfortunately in many juristictions seeding is copyright infringement and leeching is not. Seeding by default sadly gets a lot of newbs into trouble - especially as publicising IP addresses are part of the protocol. Porn blackmail companies and MPAA agents know that seeders are low hanging fruit. Similarly Limewire and ilk using the downloads folder as a default share folder is useful for the health of the network but this has led many to be unwitting uploaders - which is what they got done for. Jammie Thomas is a case in point, newb music fan (or her kids I recall) but the sharing by default is what she was convicted of - for $30,000,000. No one has ever been convicted of downloading alone - they don't bother trying. So seeding by default can be very cruel - sadly. Even with that caveat, those like Jammie, brought up on Sesame Street were taught to share and don't know how severe tne penalties can be. That sharing is or can be wrong is now taught at a nursery level.
- lolidaisuki 10y ago
  Sounds like the technology is bad laws in those jurisdictions, not technology. Thus it should be solved by changing the law(s) instead of changing the technology that's perfectly fine.
- deleted 10y ago
  [deleted]
- claudius 10y ago
  > Unfortunately in many juristictions seeding is copyright infringement and leeching is not. At least in Germany, both leeching and seeding are copyright infringements as soon as you upload any data back into the swarm. Since leeching also does this (though not exclusively), it is also copyright infringement. Pure downloading is not, though, which is why streaming websites (just downloading, no uploads) are fairly popular here.
  - deepnet 10y ago
    Yes by leech only I meant downloading alone. Well clarified thanks.
- branchless 10y ago
  A regular person hit for 30 mm. USA is insane.
- amelius 10y ago
  This is why bittorrent should have included passwords. If everybody uses the same password (e.g., "cyberpunk"), then everybody could download each other's files, while the liability would lie purely with the leecher (because he broke into someone else's machine by guessing the password).
- loup-vaillant 10y ago
  > That sharing is or can be wrong is now taught at a nursery level. Really? This is yet another step closer to "The Right to Read". Such a waste. We should end scarcity, not create it artificially.
KMag 10y ago
Former LimeWire engineer here. I implemented randomized chunk selection for swarmed downloads in LimeWire. I think the proper swarmed streaming solution is to make the percentage chance of requesting the rarest chunk be a smooth function of the number of replicas of the rarest chunk. If the rarest chunk has only one source, the probability should be 1.0. If there are multiple chunks that are all the most rare, you probably want to randomly select which chunk to request next, with an approximately exponential distribution rather than uniform distribution. Y probability of selecting the first chunk, Y^2 for the second, Y^3 for the third, etc. You'd want to run some simulations to fine-tune the probability function and also the Y percentage. What I did for LimeWire was (1) if the MIME type wasn't on a streamable whitelist, download all chunks in randomized order. (2) if the file was a streming type and less than 10% complete, start downloading the available chunk closest to the front of the file that isn't currently in progress (3) if the file is a streamable type and 10% to N% complete, randomly select either in-order or randomized selection with probability X. (4) beyond N% complete, always use random chunk selection. I'm pretty sure N was 50 and nearly certain X was 0.5. I originally proposed making X a smooth function of the % downloaded instead of 0, 0.5. 1.0 stair-steps, but the lead developer strongly preferred stair-steps. The random selection algorithm actually tried to keep the number of ranges of bytes (extents) below 5. So even after 50% downloaded, you still have a 25% chance of getting in-order downloading. The reason I used randomization instead of rarest-first was that it was my first change to LimeWire, and this was the least invasive change to make. At that time, LimeWire had a global list of verified downloaded chunks and a global list of in-progress chunks, but no global counter for number of replicas.
- KMag 10y ago
  Oh, and if the user was idle more than something like 5, 15, or 30 minutes, LimeWire would switch to random chunk requests regardless of MIME type, assuming the user didn't need a streaming download. I hope full-screen media players prevented the user from being counted as idle. I vaguely seem to remember WMV and ASF also needing some information from footers in the file, and therefor also prioritizing the last MB of the file. This was all implemented using the Strategy object oriented design pattern, to make it easier to play around with many alternatives and make specialized strategies for specific MIME types.
deleted 10y ago
[deleted]
odonnellryan 10y ago
> They still got too much heat for adding that not very usefull feature. I don't get this. It is definitely useful. I'm not saying don't upload: the client still seeds. It just downloads the contents in order.
- MoSal 10y ago
  It wasn't very useful because it didn't actually force streaming. The "rarest-first" rule was still respected. It wasn't optimal for the swarm. Yet, the user didn't really get what he/she wanted.
amadvance 10y ago
Can the seeders enforce the rarest-first approach giving priority at lechers that ask for the rarest chunks ?

by MoSal 10y ago

catwell 10y ago

This could be true, however I think that if all this torrent streaming sticks to same behavior such as that of XBMC torrent [1] (i.e. streaming for the whole duration of the movie) the swarm will end up stronger, not weaker. [1] https://github.com/steeve/xbmctorrent/blob/e6bcb1037668959e1e3cb5ba8cf3e379c6638da9/README.md#doesnt-sequential-download-on-bittorrent-is-bad https://github.com/steeve/xbmctorrent/blob/e6bcb1037668959e1...

MoSal 10y ago
Suppose there is only one seeder and 2-3 leechers. All leechers managed to get 50+%. Now, if the seeder disappears, the leechers should be able to finish the download from each other. If all initial leechers are streamers, that download will never finish for them or anyone else. Supposedly long seed intervals will not help the swarm in that case.
- RyanZAG 10y ago
  Streaming isn't going to work there anyway. In reality, there are 100+ seeders and if you had a constant supply of streamers who spend half their time seeding (assuming they download the whole movie by the 50% mark), then you have a very very healthy swarm.

deepnet 10y ago

Unfortunately in many juristictions seeding is copyright infringement and leeching is not. Seeding by default sadly gets a lot of newbs into trouble - especially as publicising IP addresses are part of the protocol. Porn blackmail companies and MPAA agents know that seeders are low hanging fruit. Similarly Limewire and ilk using the downloads folder as a default share folder is useful for the health of the network but this has led many to be unwitting uploaders - which is what they got done for. Jammie Thomas is a case in point, newb music fan (or her kids I recall) but the sharing by default is what she was convicted of - for $30,000,000. No one has ever been convicted of downloading alone - they don't bother trying. So seeding by default can be very cruel - sadly. Even with that caveat, those like Jammie, brought up on Sesame Street were taught to share and don't know how severe tne penalties can be. That sharing is or can be wrong is now taught at a nursery level.

lolidaisuki 10y ago
Sounds like the technology is bad laws in those jurisdictions, not technology. Thus it should be solved by changing the law(s) instead of changing the technology that's perfectly fine.
deleted 10y ago
[deleted]
claudius 10y ago
> Unfortunately in many juristictions seeding is copyright infringement and leeching is not. At least in Germany, both leeching and seeding are copyright infringements as soon as you upload any data back into the swarm. Since leeching also does this (though not exclusively), it is also copyright infringement. Pure downloading is not, though, which is why streaming websites (just downloading, no uploads) are fairly popular here.
- deepnet 10y ago
  Yes by leech only I meant downloading alone. Well clarified thanks.
branchless 10y ago
A regular person hit for 30 mm. USA is insane.
amelius 10y ago
This is why bittorrent should have included passwords. If everybody uses the same password (e.g., "cyberpunk"), then everybody could download each other's files, while the liability would lie purely with the leecher (because he broke into someone else's machine by guessing the password).
loup-vaillant 10y ago
> That sharing is or can be wrong is now taught at a nursery level. Really? This is yet another step closer to "The Right to Read". Such a waste. We should end scarcity, not create it artificially.

KMag 10y ago

Former LimeWire engineer here. I implemented randomized chunk selection for swarmed downloads in LimeWire. I think the proper swarmed streaming solution is to make the percentage chance of requesting the rarest chunk be a smooth function of the number of replicas of the rarest chunk. If the rarest chunk has only one source, the probability should be 1.0. If there are multiple chunks that are all the most rare, you probably want to randomly select which chunk to request next, with an approximately exponential distribution rather than uniform distribution. Y probability of selecting the first chunk, Y^2 for the second, Y^3 for the third, etc. You'd want to run some simulations to fine-tune the probability function and also the Y percentage. What I did for LimeWire was (1) if the MIME type wasn't on a streamable whitelist, download all chunks in randomized order. (2) if the file was a streming type and less than 10% complete, start downloading the available chunk closest to the front of the file that isn't currently in progress (3) if the file is a streamable type and 10% to N% complete, randomly select either in-order or randomized selection with probability X. (4) beyond N% complete, always use random chunk selection. I'm pretty sure N was 50 and nearly certain X was 0.5. I originally proposed making X a smooth function of the % downloaded instead of 0, 0.5. 1.0 stair-steps, but the lead developer strongly preferred stair-steps. The random selection algorithm actually tried to keep the number of ranges of bytes (extents) below 5. So even after 50% downloaded, you still have a 25% chance of getting in-order downloading. The reason I used randomization instead of rarest-first was that it was my first change to LimeWire, and this was the least invasive change to make. At that time, LimeWire had a global list of verified downloaded chunks and a global list of in-progress chunks, but no global counter for number of replicas.

KMag 10y ago
Oh, and if the user was idle more than something like 5, 15, or 30 minutes, LimeWire would switch to random chunk requests regardless of MIME type, assuming the user didn't need a streaming download. I hope full-screen media players prevented the user from being counted as idle. I vaguely seem to remember WMV and ASF also needing some information from footers in the file, and therefor also prioritizing the last MB of the file. This was all implemented using the Strategy object oriented design pattern, to make it easier to play around with many alternatives and make specialized strategies for specific MIME types.

deleted 10y ago

[deleted]

odonnellryan 10y ago

> They still got too much heat for adding that not very usefull feature. I don't get this. It is definitely useful. I'm not saying don't upload: the client still seeds. It just downloads the contents in order.

MoSal 10y ago
It wasn't very useful because it didn't actually force streaming. The "rarest-first" rule was still respected. It wasn't optimal for the swarm. Yet, the user didn't really get what he/she wanted.

amadvance 10y ago

Can the seeders enforce the rarest-first approach giving priority at lechers that ask for the rarest chunks ?