I dont understand why they couldnt disable that feature for only companies that wanted it (through robots.txt) or redacted it some and gave the user to drop tah

I dont understand why they couldnt disable that feature for only companies that wanted it (through robots.txt) or redacted it some and gave the user to drop taht website from your search.