Sure, the web is open you can choose to run your website the way you want it, if you don't want people linking to your content you should stop them. But the bu

Sure, the web is open you can choose to run your website the way you want it, if you don't want people linking to your content you should stop them. But the burden should be placed on the website owner and not those who are linking to the content, this isn't something like stealing content that we are talking about (which should be despised) it is simply sharing content with others no different than telling someone to check out a book named Fill in the blank. Robots.txt is great for something that is automated like scraping but otherwise it is severely limited.