3 ms·
Some services ignore global disallow, but will respect rules explicitly targeted at them.
by efreak 4y ago
Some services ignore global disallow, but will respect rules explicitly targeted at them.
- boxed 4y agoI've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.