5 ms·

Still no parallel download support ? https://bugs.archlinux.org/task/20056 https://bugs.archlinux.org/task/20056

by simula67 11y ago

Svenstaro 11y ago
Why isn't a XferCommand that by itself downloads chunks concurrently good enough for you?
- simula67 11y ago
  There are multiple ways to do it, but the defaults matter.
- broodbucket 11y ago
  XferCommands are confusing to write. I have one, and it makes my downloads faster, though I wouldn't have a clue how to write it myself. Here's what I have currently: XferCommand = /usr/bin/printf 'Downloading ' && echo %u | awk -F/ '{printf $NF}' && printf '...' && /usr/bin/aria2c -q --allow-overwrite=true -c --file-allocation=none --log-level=error -m2 --max-connection-per-server=2 --max-file-not-found=5 --min-split-size=5M --no-conf --remote-time=true --summary-interval=0 -t5 -d / -o %o %u && echo ' Complete!' It's fast, but doesn't have download progress. I had another one, which printed several lines of output per file (but did have progress!). My point is, the fact that you can make pacman download stuff quickly as-is, doesn't mean that inbuilt support wouldn't be beneficial for everyone who doesn't know how their downloading program works back-to-front.
- codyps 11y ago
  iirc, apt has support for it (it will only download a single thing at a time from a given host, but does appear to download different things from different hosts at the same time). I believe gentoo's emerge also has support (though I'm not as sure here as I typically have the downloads happen in the background while building). The overall goal here is to more effectively utilize the available bandwidth on the client side even when there are limitations a given server. It's great that we can customize how to download 1 package at a time. But there is room for improvement.
angvp 11y ago
As Dan (toofishes) would say, "patches welcome" :)
GigabyteCoin 11y ago
That would be nice, but perhaps a bit rough on the free CDNs who are nice enough to host the packages?
- codyps 11y ago
  I don't think it would be. Parallel downloads would simply mean that more of the client's bandwidth is consumed by connecting to multiple mirrors at once. Each individual mirror will, at most, still supply the same total bytes and bandwidth (though it's likely that being able to download form multiple mirrors at once would end up lowering an individual mirror's overall load by more effectively spreading downloaders between different mirrors)
anc84 11y ago
Do you really need that? I max out any bandwidth I throw at good mirrors. Only the database syncing could be faster.
laumars 11y ago
Parallel downloads are available, albeit via a 3rd party tool, `Powerpill` [1] [1] https://wiki.archlinux.org/index.php/powerpill https://wiki.archlinux.org/index.php/powerpill Personally though, I've never had an issue with the performance of `pacman`. Even without parallel downloads I've found `pacman` generally outperforms most other package management tools I've used.
- zipperhead 11y ago
  I just do this in an overnight cron: `pacman -Syuw --noconfirm` It pulls down all the updated packages but doesn't install. The packages are then ready for update when you are.
Esau 11y ago
I don't see the need for it.

Still no parallel download support ? https://bugs.archlinux.org/task/20056 https://bugs.archlinux.org/task/20056

Svenstaro 11y ago

Why isn't a XferCommand that by itself downloads chunks concurrently good enough for you?

simula67 11y ago
There are multiple ways to do it, but the defaults matter.
broodbucket 11y ago
XferCommands are confusing to write. I have one, and it makes my downloads faster, though I wouldn't have a clue how to write it myself. Here's what I have currently: XferCommand = /usr/bin/printf 'Downloading ' && echo %u | awk -F/ '{printf $NF}' && printf '...' && /usr/bin/aria2c -q --allow-overwrite=true -c --file-allocation=none --log-level=error -m2 --max-connection-per-server=2 --max-file-not-found=5 --min-split-size=5M --no-conf --remote-time=true --summary-interval=0 -t5 -d / -o %o %u && echo ' Complete!' It's fast, but doesn't have download progress. I had another one, which printed several lines of output per file (but did have progress!). My point is, the fact that you can make pacman download stuff quickly as-is, doesn't mean that inbuilt support wouldn't be beneficial for everyone who doesn't know how their downloading program works back-to-front.
codyps 11y ago
iirc, apt has support for it (it will only download a single thing at a time from a given host, but does appear to download different things from different hosts at the same time). I believe gentoo's emerge also has support (though I'm not as sure here as I typically have the downloads happen in the background while building). The overall goal here is to more effectively utilize the available bandwidth on the client side even when there are limitations a given server. It's great that we can customize how to download 1 package at a time. But there is room for improvement.

angvp 11y ago

As Dan (toofishes) would say, "patches welcome" :)

GigabyteCoin 11y ago

That would be nice, but perhaps a bit rough on the free CDNs who are nice enough to host the packages?

codyps 11y ago
I don't think it would be. Parallel downloads would simply mean that more of the client's bandwidth is consumed by connecting to multiple mirrors at once. Each individual mirror will, at most, still supply the same total bytes and bandwidth (though it's likely that being able to download form multiple mirrors at once would end up lowering an individual mirror's overall load by more effectively spreading downloaders between different mirrors)

anc84 11y ago

Do you really need that? I max out any bandwidth I throw at good mirrors. Only the database syncing could be faster.

laumars 11y ago

Parallel downloads are available, albeit via a 3rd party tool, `Powerpill` [1] [1] https://wiki.archlinux.org/index.php/powerpill https://wiki.archlinux.org/index.php/powerpill Personally though, I've never had an issue with the performance of `pacman`. Even without parallel downloads I've found `pacman` generally outperforms most other package management tools I've used.

zipperhead 11y ago
I just do this in an overnight cron: `pacman -Syuw --noconfirm` It pulls down all the updated packages but doesn't install. The packages are then ready for update when you are.

Esau 11y ago

I don't see the need for it.