4 ms·
Very cool. Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in th
by ajxs 7y ago
Very cool.
Have you considered incorporating an option for following links within the same domain to a certain depth? I remember using tools such as this in the past to save all the content from certain websites.
- flatroze 7y agoThank you! I'll add it as an issue, since it could definitely be useful for "archiving" certain resources more than 1 level deep. Do you remember the name of that tool by any chance?
- xtreme 7y agoI have used HTTrack in the past for this.
- ajxs 7y agoI wish I could remember the exact tool, this was over a decade ago. If you just do a quick internet search for such a tool you'll likely find whatever I used, it certainly wasn't anything sophisticated. It was a Windows GUI tool designed specifically for the task. Something makes me think that 'GetRight' tool might have been able to do the same thing, but I can't seem to see the feature on their website.
- flatroze 7y agoAh, I remember using something like that. I thought that tool was saving it into one .html file, but data URLs didn't exist back then, so creating directories alongside with HTML files was the only option to "replicate" a web resource, now I understand exactly what you were talking about. I'll do some more digging around and implement that in the nearest future. I may need to make all the requests async first to make sure that saving one resource with decent depth won't take too long.
- olakeasseillo 7y agoyes, "teleport pro" for win98, you could scrape a site and duplicate locally or scrape only for specific file type or size, had recursive link follow depth option and created several threads for the requests(sniff)
- darrenf 7y agoGNU wget supports recursive downloads: https://www.gnu.org/software/wget/manual/html_node/Recursive-Download.html#Recursive-Download https://www.gnu.org/software/wget/manual/html_node/Recursive...
- rmetzler 7y agoIIRC this was the mode Snowden used to bulk-download the NSA data.
- jp_sc 7y agoSiteSucker for macOS does it.
- flatroze 7y agoWell, at least it's not called iSuck.