> The old style bots were rarely a problem. They respected robot exclusions and "nofollow" warnings. What year are they reminiscing about here, 1999? Nothing h

> The old style bots were rarely a problem. They respected robot exclusions and "nofollow" warnings. What year are they reminiscing about here, 1999? Nothing has respected robots.txt in over twenty years. Nofollow isn't an anti-bot measure; it's supposed to inform search engines that you don't vouch for the linked content (don't wish to boost its rank). Nofollow doesn't mean "you must not follow this if you're a crawler".