robots.txt is not an official standard by any governmental standards bodies. This is because HTTP is a global protocol. Crawlers can and do ignore this file an

robots.txt is not an official standard by any governmental standards bodies. This is because HTTP is a global protocol. Crawlers can and do ignore this file and its contents. It's much like a post-it that says 'no peeking in these specific places', giving away the location of sensitive information you don't want crawled. It's a well-intentioned initiative but cannot in any way prevent abuse.