Good point. Now I think about it Google is actively contributing to that with their Reddit deal which has them blocking all scrapers besides Googlebot, even kno

Good point. Now I think about it Google is actively contributing to that with their Reddit deal which has them blocking all scrapers besides Googlebot, even known legit search engine crawlers like Bingbot.