3 ms·

> By using Workers to do this, it resulted in Readme's Cloudflare account receiving requests from our domain with "googlebot" useragent, but from an IP that was

by dom96 4y ago

> By using Workers to do this, it resulted in Readme's Cloudflare account receiving requests from our domain with "googlebot" useragent, but from an IP that wasn't verified as a googlebot IP address (I assume the Worker was requesting the Readme site using the Googlebot user agent but with whatever IP address is used when using CF Workers). Was this definitely the cause? It's somewhat surprising to hear that requests would be rejected if the user agent doesn't match a set of hard coded IP addresses. Were you able to resolve this in the end? If not and the cause is what you suspect then perhaps changing the user agent in your worker might be a workaround.

deleted 4y ago
[deleted]
kentonv 4y ago
It actually makes sense to me. I've pinged the bots team to see if we can improve here.
traek 4y ago
> It's somewhat surprising to hear that requests would be rejected if the user agent doesn't match a set of hard coded IP addresses. It’s fairly common for DDoS/scraping prevention, Googlebot (and most other crawlers) publish their IP ranges for that reason[0][1][2]. I don’t work at Cloudflare though, so no insider knowledge of what you folks are doing. [0] https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic https://developers.google.com/search/docs/crawling-indexing/... [1] https://developers.facebook.com/docs/sharing/webmasters/crawler/#identify https://developers.facebook.com/docs/sharing/webmasters/craw... [2] https://developer.twitter.com/en/docs/twitter-for-websites/cards/guides/troubleshooting-cards#validate_twitterbot https://developer.twitter.com/en/docs/twitter-for-websites/c...
cj 4y ago
> Was this definitely the cause? No, I haven't confirmed it. We jumped straight to fixing it without debugging the root cause. It's possible the cause is something totally different (I should have added this caveat in my original post). I was just speculating.

by dom96 4y ago

deleted 4y ago

[deleted]

kentonv 4y ago

It actually makes sense to me. I've pinged the bots team to see if we can improve here.

traek 4y ago

> It's somewhat surprising to hear that requests would be rejected if the user agent doesn't match a set of hard coded IP addresses. It’s fairly common for DDoS/scraping prevention, Googlebot (and most other crawlers) publish their IP ranges for that reason[0][1][2]. I don’t work at Cloudflare though, so no insider knowledge of what you folks are doing. [0] https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic https://developers.google.com/search/docs/crawling-indexing/... [1] https://developers.facebook.com/docs/sharing/webmasters/crawler/#identify https://developers.facebook.com/docs/sharing/webmasters/craw... [2] https://developer.twitter.com/en/docs/twitter-for-websites/cards/guides/troubleshooting-cards#validate_twitterbot https://developer.twitter.com/en/docs/twitter-for-websites/c...

cj 4y ago

> Was this definitely the cause? No, I haven't confirmed it. We jumped straight to fixing it without debugging the root cause. It's possible the cause is something totally different (I should have added this caveat in my original post). I was just speculating.