A good client will send a distinguishable user agent because the server may want to reject requests from that client. There is no robots.txt equivalent for user

A good client will send a distinguishable user agent because the server may want to reject requests from that client. There is no robots.txt equivalent for user-initiated clients, so user agents are the appropriate way to handle the situation. It is unlikely that you will want to block a traditional web browser, but web browsers are not the only clients available and some will be performing operations that are not necessarily in line with the wishes of your resources.