2 cents, With so many bots, a lot of sites run a whitelist process of user agents, so every new entrant is automatically blocked in those cases, and for the re

2 cents, With so many bots, a lot of sites run a whitelist process of user agents, so every new entrant is automatically blocked in those cases, and for the rest, any large scale crawling is quite likely to get blocked. Anecdotally it seems a relatively new search engine entrant is using Chrome user agents and follows Googlebot robots.txt directives so are impossible to block. If true, that's an absurd way for a new player to enter the market and just annoys webmasters. A recent anti-competitive report by the UK government estimated that it would take about $20bn dollars for a new entrant to compete on Google's level. Then there's how Google pay Mozilla/Apple very large sums of money based on their ROI/search which no one else can compete with, partly due to the invasive surveillance capitalism that Google follows. That ultimately means that anyone who wants to compete has to pay browsers and operating systems to enter the market in a substantial way. All that said, encouraging people to find alternative sources of information is most likely the best route to diversification with or without government(s) intervention.