Using robots.txt to avoid search engines indexing the page is the most stupid thing you can do. Not only it's not mandated by law that search engines have to fo

Using robots.txt to avoid search engines indexing the page is the most stupid thing you can do. Not only it's not mandated by law that search engines have to follow the rules in the file, but also you are giving to the public a known file where you put all the URL that you don't want to be public. And everyone that wants to get some information on a site the first thing that goes to see is the robots file. The correct thing would be to serve pages with the appropriate HTTP header to disable indexing. Of course search engines are still not obliged to follow the header, just as they are not obliged to follow the robots.txt file, but you are not leaking more information that you need. Really, robots.txt file is only useful to reduce the load on the server by crawlers, it shouldn't be used as a protective measure!