There's a non standard Noindex: /URL that Google accepts in robots.txt. Unfortunately it breaks many other crawlers which don't understand it, so you have to re

There's a non standard Noindex: /URL that Google accepts in robots.txt. Unfortunately it breaks many other crawlers which don't understand it, so you have to rely on user agent sniffing.