> If sites don't want to be found and indexed why go to any effort to include them? The news sites etc. want to be indexed by Google, and they whitelist Googl

> If sites don't want to be found and indexed why go to any effort to include them? The news sites etc. want to be indexed by Google, and they whitelist Google so they show up in search results. They don't want to be crawled by random aggregators and scrapers which don't bring them ad views and subscriptions. Indexing for me and not for thee. Other places like Slack channels are private or semi-private and shouldn't be indexed, I agree. There was some kind of scheme a while back (before Freenode went down the tubes in a completely different and unrelated way) to index Freenode. People rightfully freaked out about it and the scheme was shot down.