3 ms·
> And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header. Well I had to check, but this is not true at all:
by leni536 12y ago
> And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header.
Well I had to check, but this is not true at all:
$ nc -l -p 9999
GET / HTTP/1.1
User-Agent: Wget/1.15 (linux-gnu)
Accept: */*
Host: localhost:9999
Connection: Keep-Alive
Also wget can recursively mirror webpages and there are nice options to carefully select contents you want to download. It's quite dated though, I wish it could use an external downloader (like aria2) and only do the walking and converting links part itself.
- 72deluxe 12y agoThat mirroring option in wget has been very useful over the years, even more so if you remember to add --no-parent, else you'll end up with the ENTIRE site.
- baldfat 12y agoI love aria2c I use it all the time even as the down loader for ArchLinux's pacman (package manager).
- heyalexej 12y agoIt's also used internally in apt-fast[1] and speeds up the process by an order of magnitude for me. [1] https://github.com/ilikenwf/apt-fast https://github.com/ilikenwf/apt-fast
- 10dentious 12y agomultiple connection/source downloads is the most significant feature. Great program.
- sanqui 12y agoDo note that just a plain wget command isn't enough to properly archive/"back up" a webpage, because it doesn't save headers and other important metadata. Make sure to always use WARCs, which preserve metadata and have wonderful support, including the possibility to get absorbed into the Wayback Machine. More info: http://www.archiveteam.org/index.php?title=Wget_with_WARC_output http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou... http://www.archiveteam.org/index.php?title=The_WARC_Ecosystem http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...
- ams6110 12y agowget to make a static copy of a site is the one thing I use it for. It's awesome. I tend to reach for curl first in most other cases, for one thing it's included in the "minimal" RHEL installation so I can pretty much depend on it always being there, whereas wget is extra.
- nailer 12y agoSorry I stand corrected. I stumbled upon this info when writing a redirector app recently. You're right, I should have verified that it was true.