2 ms·
How do you know it's facebook's crawler if it going through residential proxies?
by smallerize 19d ago
How do you know it's facebook's crawler if it going through residential proxies?
- flyingcoder 19d agoIt had meta in the user-agent headers, it’s trivial to spoof but I’ve read of many others who are seeing the same thing. It’s not just them though, it’s also parallel web systems, and then a bunch of others who aren’t identifying themselves.
- d3Xt3r 19d agoThat's probably not Meta themselves, but their new agent called Muse. It's likely coming from actual individuals. Not sure why they're scraping your website though. Maybe time to turn off all the SEO stuff to make your site less discoverable?