2 ms·
wget --recursive --page-requisites --adjust-extension --convert-links --no-parent https://YOURWEBPAGEHEREX.com https://YOURWEBPAGEHEREX.com NO "--convert-links
by nix23 5y ago
wget --recursive --page-requisites --adjust-extension --convert-links --no-parent https://YOURWEBPAGEHEREX.com https://YOURWEBPAGEHEREX.com
NO "--convert-links" if you want a "pure" non local browsable copy.
- myself248 5y agoYes yes fine, and then I get throttled to 2 bytes/sec by the server. So I did some user-agent hijinks and set my delay to like 5000msec and that helped for a while, but my machine crashed and when I went to resume the task I was throttled again.
- nix23 5y ago>but my machine crashed Maybe it's not the servers who throttle you then ;)
- traverseda 5y agoWget will exhaust all available ram on a long enough crawl.
- nix23 5y agoNo, i crawled many multi-gigabyte sites with my raspberry2 for days.
- traverseda 5y agoI've had memory exhaustion (on a 4GB system) after I think about 600GB in a single crawl. Splitting it into multiple crawls is of course better. That was a site specifically set up to deal with large collections of files though.