So you are saying that each search result page are handwritten by Google editors? Of course they are not, they are simply re-published snippets of the websites

So you are saying that each search result page are handwritten by Google editors? Of course they are not, they are simply re-published snippets of the websites along and Google surrounds the results with ads. While it's a symbiotic relationship that most websites want - sharing their content for placement in Google's webpages - it's not necessarily universal and ROBOTS.TXT is hardly a "contract" covering your data's usage.