6 ms·

Downloading from several mirrors at once makes sense, but using "download accelerators" to cheat on TCP congestion control is just wrong. Some mirrors will even

Downloading from several mirrors at once makes sense, but using "download accelerators" to cheat on TCP congestion control is just wrong. Some mirrors will even ban you for making more than 4 connections at once.

k0pernicus 10y ago
Hi, actually, this was an idea that we shared a few days ago, and I am agree with this. As I mentioned earlier in a next comment, Snatch is a side project that has been created for a presentation of Rust. Thank you for this comment - we will work on this idea soon :-)
- zkms 10y ago
  Have you considered adding support for download-accelerating strategies like https://news.ycombinator.com/item?id=11842517 https://news.ycombinator.com/item?id=11842517?
the8472 10y ago
Indeed, but TCP's default congestion control might be suboptimal on some pipes. Maybe an option to choose an alternative controller via socket options (linux and windows offer alternatives) would make sense in some situations.
- ycmbntrthrwaway 10y ago
  > TCP's default congestion control might be suboptimal on some pipes Default TCP congestion control is Reno, but Linux uses Cubic, increased initial CWND and all the other modern modifications. CDNs have their own proprietary congestion controls that are arguably better, but Google has already pushed its own congestion control (BBR) into Linux, so it is just a matter of time until it becomes new default. There is no reason to tamper with these settings unless you are operating Netflix/YouTube, some sort of CDN or unusual (satellite) link. > Maybe an option to choose an alternative controller via socket options (linux and windows offer alternatives) would make sense in some situations. Only the server congestion control matters. Client simply acknowledges whatever it receives as soon as possible and waits for more data.
  - paulddraper 10y ago
    > Only the server congestion control matters. Client simply acknowledges whatever it receives as soon as possible and waits for more data. To be clear, that's true if the server is the only one transmitting data. (I.e. for a download)
gtdawg 10y ago
Some users can have more than one external IPv4/IPv6 address (One for each Internet connection, if they have multiple). In order to use all internet connections for the download, they would have to initiate multiple tcp sockets. What kind of logic should this app use to determine how many external addresses are actually available in the case of IPv4? As you say, it is more "tcp fair" to have only one TCP connection per client/server relationship in normal cases. Just some random thoughts. Apologies if it's difficult to follow. I can reply to any questions if anything is confusing.
- ycmbntrthrwaway 10y ago
  > In order to use all internet connections for the download, they would have to initiate multiple tcp sockets. Opening one connection per interface is still a violation of TCP design. Look into MPTCP RFCs. It is specifically designed to use multiple connections as if you had only one. Developers of the standard made sure you don't have any advantage compared to someone using only one connection. Two connections over two different interfaces will meet somewhere on the bottleneck near the mirror so you still have the advantage of multiple TCP connections. Sure MPTCP is not ready yet, but multipath QUIC will be deployed by the end of this year so all your Chrome downloads from google drive will utilize multiple connections.
milankragujevic 10y ago
Google does the banning, but they limit their download speed to some slow speed and I have a 1 Gbps server which I use to download stuff, so I often open up to 32 connections to their server just to download at a normal rate. They have the bandwidth, why do the limit the speed is beyond me. You can see this on the googlevideo.com domain, for Google Drive transcoded videos, thought not for YouTube.

k0pernicus 10y ago

Hi, actually, this was an idea that we shared a few days ago, and I am agree with this. As I mentioned earlier in a next comment, Snatch is a side project that has been created for a presentation of Rust. Thank you for this comment - we will work on this idea soon :-)

zkms 10y ago
Have you considered adding support for download-accelerating strategies like https://news.ycombinator.com/item?id=11842517 https://news.ycombinator.com/item?id=11842517?

the8472 10y ago

Indeed, but TCP's default congestion control might be suboptimal on some pipes. Maybe an option to choose an alternative controller via socket options (linux and windows offer alternatives) would make sense in some situations.

ycmbntrthrwaway 10y ago
> TCP's default congestion control might be suboptimal on some pipes Default TCP congestion control is Reno, but Linux uses Cubic, increased initial CWND and all the other modern modifications. CDNs have their own proprietary congestion controls that are arguably better, but Google has already pushed its own congestion control (BBR) into Linux, so it is just a matter of time until it becomes new default. There is no reason to tamper with these settings unless you are operating Netflix/YouTube, some sort of CDN or unusual (satellite) link. > Maybe an option to choose an alternative controller via socket options (linux and windows offer alternatives) would make sense in some situations. Only the server congestion control matters. Client simply acknowledges whatever it receives as soon as possible and waits for more data.
- paulddraper 10y ago
  > Only the server congestion control matters. Client simply acknowledges whatever it receives as soon as possible and waits for more data. To be clear, that's true if the server is the only one transmitting data. (I.e. for a download)

gtdawg 10y ago

Some users can have more than one external IPv4/IPv6 address (One for each Internet connection, if they have multiple). In order to use all internet connections for the download, they would have to initiate multiple tcp sockets. What kind of logic should this app use to determine how many external addresses are actually available in the case of IPv4? As you say, it is more "tcp fair" to have only one TCP connection per client/server relationship in normal cases. Just some random thoughts. Apologies if it's difficult to follow. I can reply to any questions if anything is confusing.

ycmbntrthrwaway 10y ago
> In order to use all internet connections for the download, they would have to initiate multiple tcp sockets. Opening one connection per interface is still a violation of TCP design. Look into MPTCP RFCs. It is specifically designed to use multiple connections as if you had only one. Developers of the standard made sure you don't have any advantage compared to someone using only one connection. Two connections over two different interfaces will meet somewhere on the bottleneck near the mirror so you still have the advantage of multiple TCP connections. Sure MPTCP is not ready yet, but multipath QUIC will be deployed by the end of this year so all your Chrome downloads from google drive will utilize multiple connections.

milankragujevic 10y ago

Google does the banning, but they limit their download speed to some slow speed and I have a 1 Gbps server which I use to download stuff, so I often open up to 32 connections to their server just to download at a normal rate. They have the bandwidth, why do the limit the speed is beyond me. You can see this on the googlevideo.com domain, for Google Drive transcoded videos, thought not for YouTube.