3 ms·
The article is correct: Bytespider does not observe robots.txt directives. I set Apache to give Bytespider 404 on anything. Not even Ahrefsbot is this poorly b
by bediger4000 3y ago
The article is correct: Bytespider does not observe robots.txt directives.
I set Apache to give Bytespider 404 on anything. Not even Ahrefsbot is this poorly behaved.