Not directly answering your question, but you don't allow search engines, you disallow. To be more clear, can disallow Google or other search engines from craw

Not directly answering your question, but you don't allow search engines, you disallow. To be more clear, can disallow Google or other search engines from crawling your content, but not from including it in their index. Google (et al) can gather information and ranking about your site from back links and other pages it crawls, but it might not actually have your page content. If the engines follow current internet politeness.