No, it is not. Cloudflare is giving it's customers what they want. They don't want all kinds of bots claiming to be search engines crawling their sites. Cloudf

No, it is not. Cloudflare is giving it's customers what they want. They don't want all kinds of bots claiming to be search engines crawling their sites. Cloudflare isn't hurting cloudflare competitors by doing this. Cloudflare isn't hurting their customers by doing this. To repeat - most websites don't want lots and lots of crawlers. They want the 2 or 3 which matter and no more, because at some point it's difficult to tell what the crawler is doing... (is it a search engine???). They aren't obliged to help search engines. Even if Cloudflare wasn't offering this, bigger customers would roll their own and do.. more or less the same thing.