about 18 months ago, our non-Google / Bing bot traffic went from single digits per cent to over 99.9% bot traffic. We tried some home-spun solutions at first,

about 18 months ago, our non-Google / Bing bot traffic went from single digits per cent to over 99.9% bot traffic. We tried some home-spun solutions at first, but eventually threw in the towel and put Cloudflare in front of all our publicly accessible pages. On a long term basis, this was probably the right move for us, but we felt forced into this. And the Cloudflare Managed Ruleset definitely blocks some legit traffic such that it requires a fair amount of manual tuning.