3 ms·

>but my machine crashed Maybe it's not the servers who throttle you then ;)

• traverseda 5y ago
  Wget will exhaust all available ram on a long enough crawl.
  • nix23 5y ago
    No, i crawled many multi-gigabyte sites with my raspberry2 for days.
    • traverseda 5y ago
      I've had memory exhaustion (on a 4GB system) after I think about 600GB in a single crawl. Splitting it into multiple crawls is of course better. That was a site specifically set up to deal with large collections of files though.