4 ms·
The Facebook crawler is hammering the internet
- bediger4000 1y agoWould a 404 or a 403 be more appropriate? What if you just want Meta crawlers to go away forever?
- mdhb 1y agoBetter yet… 402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/402 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
- jedisct1 1y agoInterestingly, they rotate user agents. A few days ago they were identifying as “FaceBot” (not "FacebookBot"). When that began to be blocked, they switched to reusing the “Facebookexternalhit” user agent they also use for redirects; one people are less likely to block.
- dmitrygr 1y agothat’s not interesting. That’s borderline fraud. Once someone gets a huge bill they’ll have standing to sue, and should.
- philipallstar 1y agoWhat's "borderline fraud"? Not fraud?
- N19PEDL2 1y agoGenuine question: how do they know the bot is from Facebook, apart from what's written in the user agent?
- deleted 1y ago[deleted]
- extraduder_ire 1y agoThey're cropped off to the side, but I assume the IPs making those requests are in a block owned by facebook.
- BoredPositron 1y agoMeta as a company always acts like the most unprofessional idiots if they feel under pressure. They have all the time and ressources to do it right and they never do.
- emot 1y agoisn't this crawler to generate previews on Facebook? they have others for training and AI stuff. oh well, one never knows with Meta... —— The facebookexternalhit/1.1 user agent you're seeing in the logs is a Facebook crawler, specifically used by Facebook’s servers to fetch content (like Open Graph metadata) when: Someone shares a link on Facebook or Messenger Facebook needs to generate a preview (title, image, description) for that URL