Or a 4th option: if, as you say, most people are doing SEO tricks to get there, why not just only index stuff that has indexing-specific metadata associated wit

Or a 4th option: if, as you say, most people are doing SEO tricks to get there, why not just only index stuff that has indexing-specific metadata associated with it. I don't see why this needs to be so different from an opt-in version of robots.txt. IMO this only becomes unclear when you're collecting user content en masse from non-technical users, say in a case like Facebook, etc. And IMO Facebook just shouldn't be passing on user-content from individual consumers to be indexed, especially if they encourage you to upload data that references other people.