I never have seen a good use-case for allowing one search engine not to index the page while not allowing another one. Neither today, nor 20yrs ago. For the Goo

I never have seen a good use-case for allowing one search engine not to index the page while not allowing another one. Neither today, nor 20yrs ago. For the Google and As(sense?) related stuff, using robots.txt for controlling their behaviour seems to me like a bad API. Given that Google offers fancy webmaster consoles to change the appearing on their site since years. Really, the only use case for distinguishing robots I ever stumbled upon was, as a client, blacklisting wget (looking at you, ArXiv.org). But then we just switch to curl or let wget ignore robots.txt, but with being nice to the server/network load.