I'm working on a search-related project. There is a paradox here. Many content producers hate Google taking their content without giving them a fair share of th

I'm working on a search-related project. There is a paradox here. Many content producers hate Google taking their content without giving them a fair share of the revenue that search generates. But they block any competitors to Google from spidering or crawling their websites, or rate-limit programmatic queries. The way things work in practice, much of the web tries to prevent any type of programmatic access though a combination of edge-tech and robots.txt policies. Content producers are addicted to Google and block competition, even though it is killing them. As Shakespeare put it: "Like rats that ravin down their proper bane, A thirsty evil; and when we drink we die."