3 ms·

You’d assume those have proper robots.txt configuration?

• boxed 4y ago
  I have a disallow all robots.txt for a production system. Have had from the beginning. Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.
  • efreak 4y ago
    Some services ignore global disallow, but will respect rules explicitly targeted at them.
    • boxed 4y ago
      I've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.