4 ms·

> an algo to find the optimal amount of parallel downloads An algo...? Most download managers I tried simply had a set amount of threads they tried to open; th

by toyg 5y ago

> an algo to find the optimal amount of parallel downloads An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare. That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.

zinekeller 5y ago
> the smartest ones will close some of those connections if they think they are being throttled, but that's rare. And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).
bityard 5y ago
> single-threaded downloads are severely inefficient in the age of multithreaded CPUs Is the CPU typically the bottleneck when downloading things from the Internet?
- toyg 5y ago
  If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.
  - Szpadel 5y ago
    but with today CPUs this becomes an issue with speeds at least 25gbps
    - KennyBlanken 5y ago
      Yeah. Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again. The Ryzen in my current desktop has memory that tops out at 25GB/sec.
- KennyBlanken 5y ago
  Nope. Starting at the NIC, it's not "single-threaded." Intel NIC drivers on most platforms service the multiple buffers using several available physical cores. Even if it were single-threaded, memory bandwidth over a decade ago was around 3GB/sec. Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends. Download "managers" that can queue up a list of downloads, schedule downloads (say for when you're asleep), automatically retry/partial-resume a big download, throttle a download to a low rate, and so on? That's useful for some folks. Edit: it's written in java. I'm laughing my ass off.
  - dspillett 5y ago
    > Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends. If there are any network bottlenecks, including at the host, that are enough to stop you saturating your own last-mile link with a single stream, that is simply not true. Or if the sending end sets per-stream limits. Downloading multiple streams from the same source can significantly speed up the process. Maybe it often doesn't, but that doesn't make it snake-oil. > memory bandwidth over a decade ago was around 3GB/sec Memory bandwidth is practically irrelevant if the main bottleneck is the remote network or something between you and it.

by toyg 5y ago

zinekeller 5y ago

> the smartest ones will close some of those connections if they think they are being throttled, but that's rare. And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).

bityard 5y ago

> single-threaded downloads are severely inefficient in the age of multithreaded CPUs Is the CPU typically the bottleneck when downloading things from the Internet?

toyg 5y ago
If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.
- Szpadel 5y ago
  but with today CPUs this becomes an issue with speeds at least 25gbps
  - KennyBlanken 5y ago
    Yeah. Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again. The Ryzen in my current desktop has memory that tops out at 25GB/sec.
KennyBlanken 5y ago
Nope. Starting at the NIC, it's not "single-threaded." Intel NIC drivers on most platforms service the multiple buffers using several available physical cores. Even if it were single-threaded, memory bandwidth over a decade ago was around 3GB/sec. Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends. Download "managers" that can queue up a list of downloads, schedule downloads (say for when you're asleep), automatically retry/partial-resume a big download, throttle a download to a low rate, and so on? That's useful for some folks. Edit: it's written in java. I'm laughing my ass off.
- dspillett 5y ago
  > Download "acceleration" is snake oil and all "acceleration" does is consume unnecessary resources on both ends. If there are any network bottlenecks, including at the host, that are enough to stop you saturating your own last-mile link with a single stream, that is simply not true. Or if the sending end sets per-stream limits. Downloading multiple streams from the same source can significantly speed up the process. Maybe it often doesn't, but that doesn't make it snake-oil. > memory bandwidth over a decade ago was around 3GB/sec Memory bandwidth is practically irrelevant if the main bottleneck is the remote network or something between you and it.