Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever No,

Are you saying that if your content doesn't have a robots.txt, then everyone is allowed to copy the content of the site with no restrictions whatsoever No, he is saying that if your content doesn't have a robots.txt, then everyone is allowed to index the content of the site with no restrictions whatsoever. After all, you are publishing it to the world. The fact that technology and automation are involved is irrelevant: the process is identical to me telling you that paper newspaper X had an article on Y in it's Jan 23rd, 2005 edition.