3 ms·
would be interested in directions on how to do this myself. like, take existing bookmarks, make a crawler. nice work, btw
by uticus 1y ago
would be interested in directions on how to do this myself.
like, take existing bookmarks, make a crawler.
nice work, btw
- saltysalt 1y agoThanks! If I have time, I should write a how-to blog.
- commandersaki 1y agoI've not written a crawler before, but did something similar. I needed to mirror websites ala `wget -r` and there doesn't seem to be a tool or library that aside from wget that does it, so I translated the wget -r algorithm by reading the source, as best as I could, in Go. It's not parallelised or anything as that looked complicated, but was handy when integrating it into a backend project that needed that functionality. Was a fun learning experience and I found it a bit of a complex project due to interpreting the links in HTML, so I imagine doing a crawler is even more difficult. Also found Go HTML parser not that great.