I think there is a long standing question about what robots.txt is for in general. In my opinion it was originally (and still is mostly) intended for crawlers.

I think there is a long standing question about what robots.txt is for in general. In my opinion it was originally (and still is mostly) intended for crawlers. It is for bots that are discovering links and following them. A search engine would be an obvious example of a crawler. These are links that even if discovered shouldn't be crawled. On the other end is user-requested URLs. Obviously a browser operated by a human shouldn't consider robots.txt. Almost as obviously, a tool subscribing to a specific iCal calendar feed shouldn't follow robots.txt because the human told it to access that URL. (I recall some service, can't remember if it was Google Calendar or Yahoo Pipes or something else that wouldn't let you subscribe to calendars blocked by robots.txt which seemed very wrong.) The URL preview use case is somewhat murky. If the user is posting a single link and expecting it to generate a preview this very much isn't crawling. It is just fetching based on a specific user request. However if the user posts a long message with multiple links this is approaching crawling that message for links to discover. Overall I think this "URL preview on social media" probably shouldn't follow robots.txt but it isn't clear to me.