Somewhere I read that a lot of websites don't even have robots.txt so I wonder does Google's crawler bots skip those websites or do they crawl them as someone s

Somewhere I read that a lot of websites don't even have robots.txt so I wonder does Google's crawler bots skip those websites or do they crawl them as someone said as a "fair use". Speaking generally about internet search engines; majority of the websites on the web want to get discovered and attract people to their whatever (web store, web community, web blog etc.) The interesting idea I was thinking about is that a big company which operates a search engine like a Microsoft (Bing), can pay for example popular websites like Reddit, NYT, WSJ to have exclusive crawling right and therefore block Google from crawling their websites and only allow Bing to crawl them. This would then spark search engine content war which could significantly weaken Google's monopoly because a lot of people would switch to Microsoft Bing only because Reddit results show up in their search queries. This would be akin to Netflix investing billions of dollars in the exclusive content which then brings them a lot of new subscribers and lot of new revenue. In another words - user acquisition.