The only thing I have ever used robots.txt for is to stop from leaking pagerank. I have a folder called /redirect/ and i exclude that folder in my robots.txt.

The only thing I have ever used robots.txt for is to stop from leaking pagerank. I have a folder called /redirect/ and i exclude that folder in my robots.txt. I then link to sites like this /redirect/?l=www.mysite.com Anything I don't want archived, I put behind a login wall.