2 ms·
403 is generally a bad way to get crawlers to go away - https://developers.google.com/search/blog/2023/02/dont-404-my-yum https://developers.google.com/search/b
by l1n 2y ago
403 is generally a bad way to get crawlers to go away - https://developers.google.com/search/blog/2023/02/dont-404-my-yum https://developers.google.com/search/blog/2023/02/dont-404-m... suggests a 500, 503, or 429 HTTP status code.
- vultour 2y agoThat article describes the exact behaviour you want from the AI crawlers. If you let them know they’re rate limited they’ll just change IP or user agent.
- buro9 2y ago> 403 is generally a bad way to get crawlers to go away Hardly... the article links says that a 403 will cause Google to stop crawling and remove content... that's the desired outcome. I'm not trying to rate limit, I'm telling them to go away.