2 ms·
... if you're specifically targeting Google. If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search
by zinekeller 5y ago
... if you're specifically targeting Google.
If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.