No, robots.txt is never related to authorization. Even suggesting it in the same post is misleading. If you have something that shouldn't be seen, password prot

No, robots.txt is never related to authorization. Even suggesting it in the same post is misleading. If you have something that shouldn't be seen, password protect it. Also, your rate-limiting idea misses the point. It just adds a few steps to the scraping process without actually changing the game. If you realize you can't hide profiles you need to tell your users that they should write a profile they wouldn't mind having seen at work, not pretend that making the door smaller helps.