That very document explains why banning robots.txt may still get you indexed or quoted by google (it's because Google indexes other sites, and they may quote or

That very document explains why banning robots.txt may still get you indexed or quoted by google (it's because Google indexes other sites, and they may quote or mention you), and then goes onto to explain they provide mechanisms to help solve that problem. The general point make by the post you responding is perfectly correct: Google provides many ways for a web site to stop their content from being indexed. That includes all the newspaper sites. And they very explicitly don't use it. This is www.news.com.au's robots.txt: https://www.news.com.au/robots.txt https://www.news.com.au/robots.txt Notice how they ban some spider called "NewsNow". But the don't ban Google.