3 ms·
What do folks think about implementing a web crawler that you can send to a website and it indexes every internal url on the site. I remember sitting down to wr
by trustfundbaby 6y ago
What do folks think about implementing a web crawler that you can send to a website and it indexes every internal url on the site. I remember sitting down to write one 100 years ago now, and finding it to be much trickier than I thought it would be.
- tsjq 6y agothat's interesting. grab only the URLs, not the content?
- trustfundbaby 6y agoright, but of course to do that, you'd have to grab the content to parse it :)