5 ms·
Curl and Cloudflare = Information leak – But why?
- chomp 3y agoNot a leak, this is how IndexNow works, if you don't want this, disable Crawler Hints.
- kayson 3y agoAre they on by default?
- tailspin2019 3y agoAs a long time CF user, not aware of Crawler Hints before now, I wanted to check this too. Docs imply that this is not on by default [0] [0] https://developers.cloudflare.com/cache/about/crawler-hints/ https://developers.cloudflare.com/cache/about/crawler-hints/
- Sukera 3y agoDoesn't look like it https://developers.cloudflare.com/cache/how-to/enable-crawler-hints/ https://developers.cloudflare.com/cache/how-to/enable-crawle...
- deleted 3y ago[deleted]
- efitz 3y agoMost browsers have safe browsing features that send unknown URLs to a cloud service for verification. I was shocked once to see an S3 presigned URL be accessed from Russia and India, when I had not shared it yet. It turned out to be Google safe browsing service (this case was several years ago and was verified with Google; I don’t know where URLs are investigated these days). Another possible culprit is browser extensions. If you give an extension permission to a URL or page content, you have no idea if or where it’s sending it. Some extensions intentionally send URLs to cloud services for benign purposes (ex: Microsoft safe browsing extension for non-MS browsers).
- jgrahamc 3y agoI imagine this is Crawler Hints (https://blog.cloudflare.com/tag/crawler-hints/ https://blog.cloudflare.com/tag/crawler-hints/). If OP has Crawler Hints enabled on their domain then Cloudflare will let search engines know about pages.
- tailspin2019 3y agoIs there any scenario where these would be enabled by default? Or is it strictly a manual opt-in?
- jgrahamc 3y agoThis is an opt-in feature that someone has to enable.
- Atlas22 3y agoApparently bing webmaster tools will tell you who submitted the URL if its through IndexNow. Might be worth a shot. If you cant find it this url may help (did not test): https://www.jezweb.com.au/incorrect-urls-in-bing-indexnow-from-cloudflare/ https://www.jezweb.com.au/incorrect-urls-in-bing-indexnow-fr...
- Jalad 3y agoIt could also be a site map[1] which is auto-generated by a static webpage generator (Hugo does this IIRC) [1]: https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap https://developers.google.com/search/docs/crawling-indexing/...
- throwawaybutwhy 3y ago> Update #3 > If you care about your unlisted URL privacy with Cloudflare, make sure you don't have the Crawler Hints feature enabled. - A good hunt, thanks and sorry! How about not using CloudFlare at all? Quoting from CF's own blog [0]: > Going forward, because the benefits are so substantial for site owners, search operators, and the environment, we plan to start defaulting Crawler Hints on for all our customers. Which by the way directly contradicts what jgrahamc stated in this thread. [0] https://blog.cloudflare.com/crawler-hints-supports-microsofts-indexnow-in-helping-users-find-new-content/ https://blog.cloudflare.com/crawler-hints-supports-microsoft...