That wouldn't work. Search and display snippets of search results have been ruled as fair use in multiple cases for their transformative value. You wouldn't hav

That wouldn't work. Search and display snippets of search results have been ruled as fair use in multiple cases for their transformative value. You wouldn't have a copyright claim and robots.txt has been upheld as an industry standard/easily implementable escape hatch to make it known that the crawler should not index the specified sites/pages. You could certainly say, "I'll keep you blocked in robots.txt (or block by IP or whatever) until you show my ads", but you are unlikely to succeed in that, and then there are still the downsides of not being listed in search results to consider.