The interesting thing with Google Groups is that Google cannot even crawl its own pages. This is because it fails to follow its own specification for "escaped_

The interesting thing with Google Groups is that Google cannot even crawl its own pages. This is because it fails to follow its own specification for "escaped_fragment" URLs. http://developers.google.com/webmasters/ajax-crawling/docs/specification http://developers.google.com/webmasters/ajax-crawling/docs/s... http://groups.google.com/robots.txt http://groups.google.com/robots.txt Google wants webmasters to "opt in" and let Google access #! URLs without using Javascript (they want to be able to use a bot). This is done by providing an alternative "escaped_fragment" URL. But Google themselves will not allow users to access Google Groups #! URLs without using Javascript. They will not provide alternative "escaped_fragment" URLs. Why not? Search for the newsgroup comp.unix.soures and you will see an error message that Googlebot was unable to access Google Groups because of its robots.txt Of course, you will also see comp.unix.sources has (fortunately) been mirrored on numerous servers elsewhere, probably well before the Google acquisition of Deja. I think Google should give these old newsgroups back to the community in their original format.