3 ms·
I have a lot of them in robots.txt as disallow /, of course. I have several getting 404 on any request whatsoever, Meta's AI crawler, Bytespider mainly, via Apa
by bediger4000 1y ago
I have a lot of them in robots.txt as disallow /, of course. I have several getting 404 on any request whatsoever, Meta's AI crawler, Bytespider mainly, via Apache httpd mod_rewrite.