That would not stop bots crawling the pages generated by the hosting software, about 90 pages for each of 1,000 projects, duplicated at 4 different sub-domains.

That would not stop bots crawling the pages generated by the hosting software, about 90 pages for each of 1,000 projects, duplicated at 4 different sub-domains. This stuff -- http://alioth.debian.org/projects/benchmarksgame/ http://alioth.debian.org/projects/benchmarksgame/ The X-Robots-Tag HTTP header does provide a way for admins to instruct bots not to index and not to crawl -- without requiring changes to upstream software.