It's only getting harder and harder for newcomers to compete. Especially with the rise of abusive AI scrapers, webmasters are increasingly hostile to new web cr

It's only getting harder and harder for newcomers to compete. Especially with the rise of abusive AI scrapers, webmasters are increasingly hostile to new web crawlers. Google and Bing get a pass because they're well-known, but any new service is fighting an uphill battle; a lot of sites will block them right off the bat. Nor does it help that some large sites (like Reddit) have inside deals with Google to feed them updates directly.