2 ms·
Bad for SEO!?! Count me in. SEO is what started fouling the web for humans in the first place. > robots.txt restrictions are routinely ignored, That's true of
by bediger4000 2y ago
Bad for SEO!?! Count me in. SEO is what started fouling the web for humans in the first place.
> robots.txt restrictions are routinely ignored,
That's true of Bytespider and a few other crawlers that aren't search engines or AI masticators. I have Apache set to give 404 to Bytespider and a few other user agents because of this.