4 ms·
wget has a mirror option that can do the crawling part for you.
by workhere-io 13y ago
wget has a mirror option that can do the crawling part for you.
- eli 13y agoYep. In my experience it tends to miss a fair bit of stuff, though. I think you'd want to be more aggressive about sniffing out files loaded by javascript or linked from CSS. Wget could probably serve as part of an MVP though. I don't think the crawler is actually that hard. The killer feature would be the execution and the UX. It should be usable by anyone.