4 ms·

"The HTTP User-Agent header was a mistake form the beginning." User-Agent spoofing was added to NCSA Mosaic in 1996. The public www was three years old and te

"The HTTP User-Agent header was a mistake form the beginning." User-Agent spoofing was added to NCSA Mosaic in 1996. The public www was three years old and text-only clients were still in widespread use https://raw.githubusercontent.com/alandipert/ncsa-mosaic/master/mosaic-spoof-agents https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas... NCSA Mosaic is the early graphical browser that begat Netscape Navigator that begat Firefox. Later came Internet Explorer, Safari, Chrome and so on What was the point of spoofing in 1996 Maybe it was just for fun, judging by the examples in the "mosaic-spoof-agents" file As a matter of practice, by default I do not send a user-agent header. I only send the minumum headers required. For me, that's almost always 1-2 for GET and 4 for POST For the vast majority of websites I have accessed,^1 this header minimisation has zero effect on the success of the HTTP request 1. For example, I have used a database of sites submitted to HN as way to test if header minimisation affects HTTP request success Generally I do not use a web browser to make HTTP requests. I perform text processing on the HTML, JSON or whatever is returned, using custom utilities. I store information in SQLite. I read this information as plain text, preferably 7-bit ASCII. I dislike UTF-8 Onlilne debates about "browser fingerprinting" always seem to focus on trying to "blend in", e.g., via "spoofing" As such, because browser continue to get more bloated with "features", online commenters argue in favor of sending more and more data points to servers that can be used to create a fingerprint instead of reducing the amount of data sent Because, according to their reasoning (or lack thereof), sending less data would "stand out" True, but it's generally easier to "spoof" a client that sends less data than one that sends more. And the number of sites that require a user-agent header is still smaller than the number that don't

Chu4eeno 4mo ago
I take it you either have a godly IP reputation, or you don't hit 90% of the current internet which is behind cloudflare, google, akamai, etc? They're pretty brutal in blocking non-browsers, I tend to have to add a lot of useless headers.
- 1vuio0pswjnm7 4mo ago
  According to their /clientrep-lookup database the IP address I am using has no reputation Blocking "non-browsers" and blocking IP addresses are two different things, so to speak The first is based on dumb heuristics and (incorrect) assumptions about behavior based on what software someone is (mistakenly) presumed to be using The second is based on past behaviour I send the minimum HTTP headers I only request what I want; I do not send requests for ads, tracking or telemetry I accept text formats, e.g., HTML, JSON, etc. No need for images, CSS, Javascript, etc.
  - 1vuio0pswjnm7 4mo ago
    https://www.akamai.com/us-en/clientrep-lookup https://www.akamai.com/us-en/clientrep-lookup IMO, there is a fundamental difference between (a) not using the data collection, surveillance and advertising-friendly software, i.e., popular graphical web browsers controlled by so-called "tech" companies that, surprise, said so-called "tech" companies and their business partners want people to use and (b) engaging in bad network behaviour, what some used to call poor "netiquette" The heuristics used to allegedly identify bad behaviour, for example, sending the "wrong" value in a User-Agent header, are beyond stupid But the false positive totals, "collateral damage", are apparently not large enough to matter https://seclists.org/nanog/2025/May/246 https://seclists.org/nanog/2025/May/246
1vuio0pswjnm7 4mo ago
HN replies sometimes (often) include statements unsupported by evidence For example, something like "90% of the current internet is behind Cloudflare, Google, Akamai, etc." The www is only subset of the internet and "90% of the internet" does not use a CDN In any event, even if "internet" means "www" there is no evidence to support this statement https://www.netcraft.com/blog/may-2025-web-server-survey https://www.netcraft.com/blog/may-2025-web-server-survey Further, being "behind Cloudflare, Google, Akamai, etc." does not necessarily mean a User-Agent header is required For example, https://www.cloudflare.com https://www.cloudflare.com does not require a UA header https://example.com https://example.com is "behind CF" and does not require a UA header https://www.google.com https://www.google.com does not require a UA header https://developers.akamai.com https://developers.akamai.com does not require a UA header And so on Individual www users, unless doing mass scanning or the like, probably do not access 100% of the www's sites (according to Netcraft, most are inactive) More likely, individuals access only a fraction of the active sites It's unlikely that individuals access exactly the same list of websites^1 For me personally, the vast majority of sites that I access do not require a UA header The sample set I use as evidence to support comments I make about this on HN is a list of all sites submitted to HN over a recent period.^2 Currently this is 12,641 sites 1. To have a meaningful discussion about UA header requirements, a sample set of sites must be agreed upon 2. Perhaps HN readers access these sites and possibly access some of the same sites. Of course it's also possible that there are HN commenters who do not actually access HN submissions and instead only read comment threads and submit replies https://news.ycombinator.com https://news.ycombinator.com does not require a UA header
1vuio0pswjnm7 4mo ago
NB. Akamai does not require any specific value for the User-Agent header nor the presence of this header For example, https://www.apple.com https://www.apple.com does not require a UA header For example, http://www.slackware.com http://www.slackware.com does not require a UA header. However, it does require Accept and Accept-Encoding headers This site can also be accessed over HTTPS

Chu4eeno 4mo ago

I take it you either have a godly IP reputation, or you don't hit 90% of the current internet which is behind cloudflare, google, akamai, etc? They're pretty brutal in blocking non-browsers, I tend to have to add a lot of useless headers.

1vuio0pswjnm7 4mo ago
According to their /clientrep-lookup database the IP address I am using has no reputation Blocking "non-browsers" and blocking IP addresses are two different things, so to speak The first is based on dumb heuristics and (incorrect) assumptions about behavior based on what software someone is (mistakenly) presumed to be using The second is based on past behaviour I send the minimum HTTP headers I only request what I want; I do not send requests for ads, tracking or telemetry I accept text formats, e.g., HTML, JSON, etc. No need for images, CSS, Javascript, etc.
- 1vuio0pswjnm7 4mo ago
  https://www.akamai.com/us-en/clientrep-lookup https://www.akamai.com/us-en/clientrep-lookup IMO, there is a fundamental difference between (a) not using the data collection, surveillance and advertising-friendly software, i.e., popular graphical web browsers controlled by so-called "tech" companies that, surprise, said so-called "tech" companies and their business partners want people to use and (b) engaging in bad network behaviour, what some used to call poor "netiquette" The heuristics used to allegedly identify bad behaviour, for example, sending the "wrong" value in a User-Agent header, are beyond stupid But the false positive totals, "collateral damage", are apparently not large enough to matter https://seclists.org/nanog/2025/May/246 https://seclists.org/nanog/2025/May/246

1vuio0pswjnm7 4mo ago

HN replies sometimes (often) include statements unsupported by evidence For example, something like "90% of the current internet is behind Cloudflare, Google, Akamai, etc." The www is only subset of the internet and "90% of the internet" does not use a CDN In any event, even if "internet" means "www" there is no evidence to support this statement https://www.netcraft.com/blog/may-2025-web-server-survey https://www.netcraft.com/blog/may-2025-web-server-survey Further, being "behind Cloudflare, Google, Akamai, etc." does not necessarily mean a User-Agent header is required For example, https://www.cloudflare.com https://www.cloudflare.com does not require a UA header https://example.com https://example.com is "behind CF" and does not require a UA header https://www.google.com https://www.google.com does not require a UA header https://developers.akamai.com https://developers.akamai.com does not require a UA header And so on Individual www users, unless doing mass scanning or the like, probably do not access 100% of the www's sites (according to Netcraft, most are inactive) More likely, individuals access only a fraction of the active sites It's unlikely that individuals access exactly the same list of websites^1 For me personally, the vast majority of sites that I access do not require a UA header The sample set I use as evidence to support comments I make about this on HN is a list of all sites submitted to HN over a recent period.^2 Currently this is 12,641 sites 1. To have a meaningful discussion about UA header requirements, a sample set of sites must be agreed upon 2. Perhaps HN readers access these sites and possibly access some of the same sites. Of course it's also possible that there are HN commenters who do not actually access HN submissions and instead only read comment threads and submit replies https://news.ycombinator.com https://news.ycombinator.com does not require a UA header

1vuio0pswjnm7 4mo ago

NB. Akamai does not require any specific value for the User-Agent header nor the presence of this header For example, https://www.apple.com https://www.apple.com does not require a UA header For example, http://www.slackware.com http://www.slackware.com does not require a UA header. However, it does require Accept and Accept-Encoding headers This site can also be accessed over HTTPS