> Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years It's not just robots.txt, it's also cloudfl

> Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years It's not just robots.txt, it's also cloudflare and IP-based throttling. And it is very, very commonplace: http://gigablast.com/blog.html http://gigablast.com/blog.html