4 ms·
Hmm. True. Though obviously crawlers can be allowed different access. Doesn't Google have a policy that sites must allow access from Google links in order to h
by waterlesscloud 11y ago
Hmm. True. Though obviously crawlers can be allowed different access.
Doesn't Google have a policy that sites must allow access from Google links in order to have content indexed? See for example the New York Times and their paywall, which is famously circumvented via Google.
That seems to indicate it's not that clear cut. My suspicion is they haven't really thought about it yet.
- shock-value 11y agoThat's not how their policy works, as far as I know. It's based on individual pages, not the entire URL.
- pests 11y agoNot directly answering your question, but you don't allow search engines, you disallow. To be more clear, can disallow Google or other search engines from crawling your content, but not from including it in their index. Google (et al) can gather information and ranking about your site from back links and other pages it crawls, but it might not actually have your page content. If the engines follow current internet politeness.