Does robots.txt not fulfill this requirement? It’s the same problem as web search. Google builds a derivative work (a mapping from queries to pages) by scraping

Does robots.txt not fulfill this requirement? It’s the same problem as web search. Google builds a derivative work (a mapping from queries to pages) by scraping the web with opt-out consent.