Here are some resources: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers https://developers.facebook.com/docs/sharing/webmasters/web-...

Here are some resources: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers https://developers.facebook.com/docs/sharing/webmasters/web-... The article states that these may bypass `robots.txt`, so you may not even bother with that. There are people who use facebook as a proxy for scraping with the preview functionality. https://datadome.co/threat-research/how-facebook-was-used-as-a-proxy-by-web-scraping-bots/ https://datadome.co/threat-research/how-facebook-was-used-as... >With technical detection only, the fraudulent requests would have been indistinguishable from legitimate API calls, since they had the Facebook user agent and IP address. However, our heuristic analysis uncovered that certain parameters, unlikely to be used by humans, were overrepresented in the URLs that Facebook requested. Certain use cases were also unlikely: for example, a human user would typically share a link to a specific ad, not to a page of search results. According to that article, looking at the requests and discerning patterns of bogus traffic would be preferable to more drastic approaches such as blocking or rate limiting indiscriminately, and risking to impact legitimate traffic.