4 ms·
No surefire way, same as blocking any other poorly-behaved crawlers. You can use robots.txt, Cloudflare, CAPTCHAs, IP range blocks, etc., but at the end of the
by solardev 2y ago
No surefire way, same as blocking any other poorly-behaved crawlers. You can use robots.txt, Cloudflare, CAPTCHAs, IP range blocks, etc., but at the end of the day none of those are foolproof. If someone really wants to scrape you, they will.
- dakiol 2y ago> If someone really wants to scrape you, they will. I don't think that's a problem. What you said is right, but no giant (like OpenAI or Google) will employ resources to scrape YOUR website. They do everything at scale, so it should be enough to block the big players (until they find a way to bypass the block, which it may take them time and money)