4 ms·
I too had same doubt. Hope someone who is expert in SEO could answer this in better way.
by techaddict009 13y ago
I too had same doubt. Hope someone who is expert in SEO could answer this in better way.
- fendmark 13y agoInteresting Yeah, Facebook's robots.txt file white-lists specific search engines and offer an option to get whitelisted ,then there is a wildcard disallow for everyone else. Of course nefarious scrapers can ignore the robots.txt file or even spoof google or bingbot, but it least it sets a precedent and a policy that they can take further action on if needed.