4 ms·
It would be awesome to see support for following links to a specified depth, similar to [Httrack](https://www.httrack.com/ https://www.httrack.com/)
by joeyhage 3y ago
It would be awesome to see support for following links to a specified depth, similar to [Httrack](https://www.httrack.com/ https://www.httrack.com/)
- codetrotter 3y agoI made a basic crawler using Firefox, thirtyfour https://docs.rs/thirtyfour/latest/thirtyfour/ https://docs.rs/thirtyfour/latest/thirtyfour/ and squid Basically, I took a start URL for the crawl, and my program would load the page in Firefox using thirtyfour, and then extract all links from the page and use some basic rules for keeping track of which ones to visit and in which order. I had Squid proxy configured to save all traffic that passed through it. It worked ok-ish. I only really stopped that project because of a hardware malfunction. The main annoyance that I didn’t get around to solving was being more smart about not trying to load non-html content that was already loaded anyway as part of the page. Because the way I extracted links from the page I also extracted URLs of JS, CSS etc that were referenced.
- gildas 3y agoYou can have a look at the last 2 examples here [1]. [1] https://github.com/gildas-lormeau/single-file-cli?tab=readme-ov-file#run https://github.com/gildas-lormeau/single-file-cli?tab=readme...