6 ms·
Just crawl, and then disregard sites that have certain scripts present.
by ErikAugust 7y ago
Just crawl, and then disregard sites that have certain scripts present.
- anon1m0us 7y agoYeah, technically, it'd be very easy. I've written countless spiders, crawlers, etc. Basically, only index items where webpage.indexOf('google') < 0