4 ms·
I was thinking about this. I'm lazy and not very talented, so is there any decent code existing and open sourced that could be adapted to spider only sites with
by perl4ever 7y ago
I was thinking about this. I'm lazy and not very talented, so is there any decent code existing and open sourced that could be adapted to spider only sites without advertising?
- ErikAugust 7y agoJust crawl, and then disregard sites that have certain scripts present.
- anon1m0us 7y agoYeah, technically, it'd be very easy. I've written countless spiders, crawlers, etc. Basically, only index items where webpage.indexOf('google') < 0