3 ms·
Are you suggesting to mirror e.g. the entire Wikipedia through wget? That is not only suboptimal, it is stressing on the server. At least you added a --wait=2,
by berkes 5y ago
Are you suggesting to mirror e.g. the entire Wikipedia through wget?
That is not only suboptimal, it is stressing on the server. At least you added a --wait=2, but on any large site/hoster/CDN, this might still get your IP banned or throttled. And on e.g. the English wikipedia this will then take 149 days. Which means that by the time you hit the last page, the first ones (and their links) are out of date.
- falcolas 5y agoIf you add '--no-parent' (doesn't request anything that's not a page dependency above the requested URI) and a '--level=5' (only follows link 5 deep), you won't get all of a site. It makes it more realistic for getting wikipedia articles.