We all know the answer: https://danluu.com/googlebot-monopoly/ https://danluu.com/googlebot-monopoly/ In my dream world, Cloudflare would make this a two way s

We all know the answer: https://danluu.com/googlebot-monopoly/ https://danluu.com/googlebot-monopoly/ In my dream world, Cloudflare would make this a two way street: you get protection against malicious bots, but you have to allow equitable treatment of all well behaved search engines (i.e. you can use robots.txt to ban lesser search engines, but doing that requires you to ban Google as well). Disclaimer: I have no current economic interest, but kinda want to run a personal search engine.