3 ms·
> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use
by danso 8y ago
> The first is that we use Aria2 to handle FTP duties. Earlier versions of the script used other FTP clients that were either slow as molasses or painful to use. After some trial and error, I found Aria2 did the job better than lftp (which is fast but fussy) or good old ftp (which is both slow and fussy). I also found some incantations that took download times from roughly an hour to less than 20 minutes.
Tangential question: is it possible to use wget for ftp duties? Though may be additional FTP-specific functionality in `aria2c` of course:
https://serverfault.com/questions/25199/using-wget-to-recursively-download-whole-ftp-directories https://serverfault.com/questions/25199/using-wget-to-recurs...
- na85 8y agoHave a look at the wget manual. There are lots of FTP-related options.
- jolmg 8y agoPerhaps he meant in their specific scenario, since he linked a page that shows use of wget for FTP.
- danso 8y agoYes, I should have specified that I was interested in what aria2 provides for FTP in addition to what a more ubiquitous tool like wget seemingly has. u/rasz says aria2 allows multi-connections, so that seems sensible: https://news.ycombinator.com/item?id=17508858 https://news.ycombinator.com/item?id=17508858
- rasz 8y agoaria is multi connection(aria2c -x5 means five concurrent), thats the main reason for speed bump
- flukus 8y agoDoes this increase speed when it's only downloading a single file at a time? It might be better off using makes multi process (make -j 5) to be able to process data while still loading other data.
- therein 8y agoEach connection requests a different range within the same file and they download together.
- deleted 8y ago[deleted]