While robots.txt is not a good security measure, it's a good tool for preventing pollution of search results with pages like wp_login.php. More to steer users t

While robots.txt is not a good security measure, it's a good tool for preventing pollution of search results with pages like wp_login.php. More to steer users to the subset of pages that are useful to them, away from the pages which are only an implementation detail. Also, some poorly rate-limited crawlers actually abide by robots.txt, so it's useful to prevent unnecessary load.