I don't think your analogy works. Racial profiling is when law enforcement decides to target someone someone based on their ethnicity (an innate characteristic

I don't think your analogy works. Racial profiling is when law enforcement decides to target someone someone based on their ethnicity (an innate characteristic) instead of their behavior. Whether or not an HTTP client sends a user-agent header is a part of its behavior when making a request. Its fair to observe that along with other aspects of the clients behavior when deciding how to handle it. You're probably being blocked by sites running WAFs like ModSecurity. There's a lot of crap out there, and I don't blame people for setting a low threshold for blocking unusual requests. Here's the rules which would catch a missing user-agent: https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/master/base_rules/modsecurity_crs_21_protocol_anomalies.conf https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/mas...