Couldn't Google just remove sites demanding special treatment from their search results? It would be right in line with their current behavior, ethically and m

Couldn't Google just remove sites demanding special treatment from their search results? It would be right in line with their current behavior, ethically and mechanically; a media company whining to the Australian government for protectionism is just a high-overhead way of disallowing all crawler traffic in robots.txt. Less pithily: Google crawls the web, offering a standard set of terms on which it does so. No website is compelled to accept these terms and have their links displayed on google.com, even if they're public-facing. The mechanism by which you can opt out, fully or partially, is a robots.txt file specifying to the crawler what it can and can't crawl and display. Australian media companies are saying that they don't accept Google's linking terms; they proposed counter-terms of licensing fees for linking, and Google can refuse them. As the two sides have failed to agree to terms on which the media sites can be linked to by Google, they fall neatly into the category of a website blocking robots.txt