It's easy for a crawler to parse a robots.txt in different ways. The article says to whitelist a few and deny everything else. The crawler could parse that, a

It's easy for a crawler to parse a robots.txt in different ways. The article says to whitelist a few and deny everything else. The crawler could parse that, and change the user agent to one whitelisted on further requests.