2 ms·

I think you're confused, here is the robots.txt from this site (linked by you) User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? user is not disa

by mariorz 18y ago

I think you're confused, here is the robots.txt from this site (linked by you) User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? user is not disallowed anywhere, vote? and reply? and x? are, but they are not content, they are actions, basically all of the content of this site is indexed. Your idea of site owners being selective about SE makes no sense to me. Would users then have to use 5 or 6 different search engines to accommodate all possible webmaster preferences?

briansmith 18y ago
You are right, I thought the discussions had /reply? URLs but they are /item?. I think that SE revenue sharing will happen. Only the SEs that share revenue will survive--a SE that doesn't share revenue will get locked out of the most valuable content and it will be forced out of business.