5 ms·

I used a lot of download managers and accelerators back in the dial-up times, they seemed like magic to me, plus being able to resume a download after somebody

by _benj 5y ago

I used a lot of download managers and accelerators back in the dial-up times, they seemed like magic to me, plus being able to resume a download after somebody picked up the phone and dropped my internet connection was a plus! How is it that they work, that is the accelerator part? My guess is an algo to find the optimal amount of parallel downloads? Edit: also with parallel downloads, is it possible to do a HTTP GET with an bytes offset? With video I guess it’s possible to select what part to stream, but with regular http download I’m not sure

charcircuit 5y ago
You want to use the Range header. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ra...
- _benj 5y ago
  so that's how `wget -c` works! the more you know :)
  - eatonphil 5y ago
    Since flags are hard to google, what does `wget -c` do?
    - loeg 5y ago
      Resume partial download.
      - eatonphil 5y ago
        Gotcha, thanks
    - elsherbini 5y ago
      My favorite way to find out what flags do is explainshell: e.g. https://explainshell.com/explain?cmd=wget+-c https://explainshell.com/explain?cmd=wget+-c
      - eatonphil 5y ago
        Wow that's a cool idea! I'll have to try it out next time.
- eatonphil 5y ago
  Point in this project's code where you can see this happen [0]. Also interesting to note that it's not just HTTP it supports but other protocols I haven't heard of like DASH, HDP, HLS [1]. [0] https://github.com/subhra74/xdm/blob/master/app/src/main/java/xdman/downloaders/http/HttpChannel.java#L123 https://github.com/subhra74/xdm/blob/master/app/src/main/jav... [1] https://github.com/subhra74/xdm/tree/master/app/src/main/java/xdman/downloaders https://github.com/subhra74/xdm/tree/master/app/src/main/jav...
  - charcircuit 5y ago
    Those protocols are on top of HTTP so it makes sense they support it too.
toyg 5y ago
> an algo to find the optimal amount of parallel downloads An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare. That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.
- zinekeller 5y ago
  > the smartest ones will close some of those connections if they think they are being throttled, but that's rare. And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).
- bityard 5y ago
  > single-threaded downloads are severely inefficient in the age of multithreaded CPUs Is the CPU typically the bottleneck when downloading things from the Internet?
  - toyg 5y ago
    If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.
    - Szpadel 5y ago
      but with today CPUs this becomes an issue with speeds at least 25gbps
      - KennyBlanken 5y ago
        Yeah. Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again. The Ryzen in my current desktop has memory that tops out at 25GB/sec.

by _benj 5y ago

charcircuit 5y ago

You want to use the Range header. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ra...

_benj 5y ago
so that's how `wget -c` works! the more you know :)
- eatonphil 5y ago
  Since flags are hard to google, what does `wget -c` do?
  - loeg 5y ago
    Resume partial download.
    - eatonphil 5y ago
      Gotcha, thanks
  - elsherbini 5y ago
    My favorite way to find out what flags do is explainshell: e.g. https://explainshell.com/explain?cmd=wget+-c https://explainshell.com/explain?cmd=wget+-c
    - eatonphil 5y ago
      Wow that's a cool idea! I'll have to try it out next time.
eatonphil 5y ago
Point in this project's code where you can see this happen [0]. Also interesting to note that it's not just HTTP it supports but other protocols I haven't heard of like DASH, HDP, HLS [1]. [0] https://github.com/subhra74/xdm/blob/master/app/src/main/java/xdman/downloaders/http/HttpChannel.java#L123 https://github.com/subhra74/xdm/blob/master/app/src/main/jav... [1] https://github.com/subhra74/xdm/tree/master/app/src/main/java/xdman/downloaders https://github.com/subhra74/xdm/tree/master/app/src/main/jav...
- charcircuit 5y ago
  Those protocols are on top of HTTP so it makes sense they support it too.

toyg 5y ago

> an algo to find the optimal amount of parallel downloads An algo...? Most download managers I tried simply had a set amount of threads they tried to open; the smartest ones will close some of those connections if they think they are being throttled, but that's rare. That's just all there is to it: single-threaded downloads are severely inefficient in the age of multithreaded CPUs (and of operating systems without a finely-tuned network stack), so by leveraging parallelization anywhere you get a chance, you multiply the speed.

zinekeller 5y ago
> the smartest ones will close some of those connections if they think they are being throttled, but that's rare. And some of them reset and restart the download, in case that the bottleneck is neither in the machine nor the server (for example, some routers will throttle long-lived connections).
bityard 5y ago
> single-threaded downloads are severely inefficient in the age of multithreaded CPUs Is the CPU typically the bottleneck when downloading things from the Internet?
- toyg 5y ago
  If the app uses a single execution thread, the OS will likely be unable to distribute work across different CPU threads. Whereas if you create multiple threads, the OS can max out CPU and I/O resources efficiently.
  - Szpadel 5y ago
    but with today CPUs this becomes an issue with speeds at least 25gbps
    - KennyBlanken 5y ago
      Yeah. Twenty year old CPUs had 1600MB/sec memory. 4-5 years later memory speed doubled. In 2010 it doubled again. The Ryzen in my current desktop has memory that tops out at 25GB/sec.