16 ms·
HTTP headers we don't want
- notyourday 8y agoThanks, Fastly. But here's the thing - you are a pipe. You maybe a pipe with some brains and ability to do request collapsing but you are still a pipe. Just like I do not care that these headers may be upsetting Comcast, I do not care that they are upsetting you. If you don't like it, there other CDNs out there. And if I need to change any of my software to make you happy, you are just not going to have business, which is probably why your sales people are giving 20-30% discounts with $500 monthly commit. That level of a discount on that level of spend is unheard in a CDN industry.
- moolcool 8y ago+1, some of the headers they deem unnecessary could be being used by the front-end or API clients (specifically `pragma` and `expires`)
- crooked-v 8y agoThe article specifically makes the point that both pragma and expires have long since been replaced by Cache-Control in anything even vaguely modern.
- notyourday 8y agoThat's what fastly does, not their customers.
- alexchamberlain 8y agoI think you may be taking the headline too seriously; I think the OP is really arguing that these headers constitute a reasonable amount of bandwidth, and maybe we should just switch them off if they're not providing any value?
- notyourday 8y agoFastly charges for bytes transferred. It also charges for going back to origin.
- venning 8y agoIn which case, continued usage of obsolete headers is to Fastly's benefit. Turning them off saves the customer money. For what it's worth, the author is listed as a "Developer Advocate".
- notyourday 8y agoNo, they are getting too many small customers and too many small customers ( see the part where now with $500/mo commit you can get 20-30% discount of the list without squeezing them hard) vary on too many headers which means it is blowing holes in their caches, which is making their Varnish-as-a-Service not work as well as it used to. So it seems they are starting to fall into the propaganda mode to paint over the issue rather than admit that it is time for them to start innovating again. Surrogate keys and quite cache busting used to be Fastly special sauce but since 2014 it is rather standard.
- cortesoft 8y agoMan, what did fastly do to you?
- matthewmacleod 8y agoThis is one of the most aggressively worthless posts I've ever read. This is literally nothing more than a minor blog post that points out that some of us are still using headers we might not need to. Finding anything else in that is utterly baffling.
- macintux 8y agoYou must have read a completely different article than I did. The one I read was providing a useful resource on obsolete/insecure/dubious but still widely-used HTTP headers. You seem to be making assumptions about the motivation for the article and then reacting strongly against it, but that's also dubious.
- matthewmacleod 8y agoDid you even read this article? A serious question, rather than a troll. It's not about what you seem to think it's about.
- deleted 8y ago[deleted]
- ShaneWilton 8y agoMost of the suggestions in this post are great, but as always, especially when security is involved, you need to assess your business needs yourself. The suggestion to use Content-Security-Policy over X-Frame-Options is great -- if you don't expect many of your users to be using IE-based browsers. If you're primarily serving large enterprises or government customers though, it's likely that most of your users will still be coming from a browser that doesn't support Content-Security-Policy.
- merb 8y agosame with Expires, if you serve really really old clients you might still need it.
- Ajedi32 8y agoBut interestingly, they deem `x-ua-compatible` "useful" even though AFAIK that's also only needed for backwards compatibility with IE.
- AbacusAvenger 8y agoNot to mention that Content-Security-Policy can be costly to set up and maintain properly. My servers send both X-Frame-Options and Content-Security-Policy, but I do keep running into cases where my CSP was too restrictive and have to keep fiddling with it.
- deleted 8y ago[deleted]
- brobinson 8y ago>P3P is a curious animal. This was a requirement to have IE6 accept third party cookies from your site.
- synhare 8y agoThis part of the article really threw me off. Someone writing an article on HTTP headers for a major CDN has never had to deal with IE6?
- barneygale 8y agoAre people really still dealing with IE6? I gave up web dev almost a decade ago and it's disturbing to hear that IE6 is still an issue!
- shakna 8y agoEnterprise. A lot of Enterprise applications are required for work, and require very old, very stubborn browsers. Most emulations don't work. You need the old stuff (IE), for the old applications to work, and as long as they can force it to, they won't update said old application. I've even had to touch systems which required IE 4 in the last few years, from before Trident became the rendering engine.
- zbentley 8y agoHealthcare and government (US). So, so very many systems are on IE6. So, so very many websites only work correctly/fully when end users are on that platform. Until you've had to support code distributed by the US federal gov't and watch the percentages of users hitting your site from XP (or earlier) UAs rise to the double digits, you have not known sadness.
- gboudrias 8y agoAlso all of China or so I was told about a year ago.
- 8y ago
- buro9 8y agoVia is not safe to remove and Fastly know this as well as Akamai, Cloudflare and others. A very cheap attack is to chain CDNs into a nice circle. This is what Via protects against: https://blog.cloudflare.com/preventing-malicious-request-loops/ https://blog.cloudflare.com/preventing-malicious-request-loo... Just because a browser doesn't use a header does not make the header superfluous.
- djhworld 8y agoVery interesting link, thanks. I'm not too familiar with this area, but from my understanding of the article, Cloudflare are suggesting that all players in the game need to be compliant otherwise nobody wins So is this Fastly article suggesting a different point of view?
- 21 8y agoWell, if one CDN doesn't respect it, it will probably soon find out :) So it's a self correcting problem.
- voidlogic 8y agoWouldn't it be better to use the "Forwarded" header for this?
- buro9 8y agoWhy better? Via is a MUST https://tools.ietf.org/html/rfc7230#section-5.7.1 https://tools.ietf.org/html/rfc7230#section-5.7.1 Forwarded is OPTIONAL https://tools.ietf.org/html/rfc7239#section-4 https://tools.ietf.org/html/rfc7239#section-4 Protecting against loops only works with MUST headers.
- justinsaccount 8y agocache-control doesn't completely replace Expires for some use cases. If you have a scheduled task that generates data every hour, you can set Expires accordingly so all clients will refresh the data as soon as the hour rolls over. You can do this using max-age but then you have to dynamically calculate this header per request which means you can't do things like upload your data to s3 and set the cache-control header on it. With expires, I can upload a file to s3 and set Expires: ... 17:00 and then not have to touch it again for an hour. you can work around this client side with per hour filenames or the other usual cache busting tricks, but that's annoying.
- AbacusAvenger 8y agoIt seems like kind of an unlikely scenario that you'd want to expire content at a specific time. I mean, if someone chooses to do that, they better know what the impact could be. With the Expires header, all clients that retrieved that content would expire at the exact same time, which could cause some disproportionately high load in the few seconds after that (the "thundering herd" problem). The Cache-Control solution will stagger the expirations (relative to when the client last retrieved it) so the server doesn't get trampled.
- AstralStorm 8y agoCongratulations, your infrastructure stability now depends on particular web browsers and caches implementing their caches and headers correctly... It takes just one big bad actor to break. Reminds me of certain routers damaging NTP traffic.
- AbacusAvenger 8y agoThat's a cynical view, and I don't think I said you should depend on Cache-Control working. Yes, there will be bad actors, but the majority of clients are good actors. It's just one of several measures you should take to even out the load. Of course you'd want a caching layer in front of the server doing the actual work, but it's still possible to "thundering herd" the cache server if you use an Expires header. Even if the herd doesn't hurt your backend server, it can still make the load on your caching frontend servers spike at specific time periods with every good actor refreshing the content at the same time. So it's still ideal to try and even out that load with Cache-Control.
- Hamuko 8y agoP3P is unnecessary until you have clients complaining that Internet Explorer users cannot use the site and it's hurting their business. I speak of experience. Curiously enough, P3P enforcement depends on the operating system and not on the browser. Internet Explorer 11 may or may not care about P3P depending if you're on Windows 7 or Windows 10.
- pfarrell 8y agoCame here to say the exact same thing. P3P may be "officially" obsolete, but if your business wants older browsers to be able to handle your code, you're going to have to deal with it. If you have the misfortune of encountering it, you can get really hard to detect bugs with ajax calls or script files not getting loaded in IE when you don't have P3P set up correctly. (for instance: https://www.techrepublic.com/blog/software-engineer/craft-a-p3p-policy-to-make-ie-behave/ https://www.techrepublic.com/blog/software-engineer/craft-a-...)
- Theodores 8y agoIt would be helpful to have a guide to this for people running a 'low audience website' where there is no CDN or Varnish, just some Apache or Nginx server on a slow-ish but cheap VPS. For a local business or community, e.g. an arts group with a Wordpress style site, there are many common problems, they might not need a full CDN, just serving media files from a cookieless subdomain gets their site up to acceptable speed cutting the header overhead considerably. Purging the useless headers might also include getting rid of pointless 'meta keywords' and what not. The tips given here could be really suited to this type of simple work to get a site vaguely performant. How to do it with common little guy server setups could really help.
- CapacitorSet 8y agoRealistically, how much traffic is saved by cutting headers? A simple article like [this](https://tp69.wordpress.com/2018/04/17/completely-silent-computer/ https://tp69.wordpress.com/2018/04/17/completely-silent-comp...) (currently on the HN frontpage) weighs 178 KB, and that's without external resources. Unused headers account at best for 0,1% of the total traffic.
- ovao 8y agoOne could argue that the headers comprise a very important 0.1%, but any wasted time the client spends waiting for and parsing headers will almost always be utterly dominated by the unavoidable wait for HTML parsing, JavaScript parsing, painting and so on. I could see the argument for pruning useless headers if, say, the method for generating them relied on some high-latency database call or filesystem access, but that would rarely be the case.
- JeremyBanks 8y agoThis grey-on-white is hard to read. I gave up on the article.
- shakna 8y agoNot for some, but for others it is so. I'm blue-black colorblind, which makes everything of that kind rather painful to view. However, the HTML is well-formed enough that a Reader-view, whichever your browser supports, should work to be able to view it.
- manigandham 8y agoIt's more of the skinny font weight, which is somewhat of an unfortunate hipster design trend that only looks good on high-res retina macbooks.
- phyzome 8y agoSaying that a header is useless because it has been deprecated and displaced by a newer header is... misleading at best. If all you ever code for is the latest version of Firefox and Chrome, you might not understand this, but there's a whole world out there with an astonishing diversity of browsers. (Also, your site is bad and you should feel bad.) Removing X-Frame-Options without first checking if 99.99% of your users' browsers support Content-Security-Policy is just asking for increased risk.
- organicmultiloc 8y agoI've worked on a lot of cross browser bugs since IE5, never seen one created by any header or created by omitting http headers. Honestly your comment is just nonsense.
- ovao 8y agoPrematurely axing X-Frame-Options would expose a security risk, not a “bug”, per se.
- tatersolid 8y agoHeaders which have caused production bugs or misbehaviors during my career (mostly in proxies and old IE): Content-Encoding Vary Cache-Control (proxies ignoring private usually) Content-Disposition X-UA-Compatible P3P Strict-Transport-Security X-Forwarded-For Via Referrer
- nebulous1 8y agoThe details are interesting but "adds overhead at a critical time in the loading of your page" ... this seems pretty unlikely to have any noticeable processing overhead. Doing things better is generally good, but this all seems very low impact.
- __jal 8y agoDepends on where you measure it. A client on a decent connection will never notice. If you're serving billions of hits, 20 bytes in a header is something you will definitely notice on your bandwidth bill.
- tgsovlerkhgsel 8y agoMore importantly, an extra header may push you into needing another packet, which creates extra potential for packet loss.
- dijit 8y agoThat's a very weak argument to make though when regarding the modern web; sites are routinely sending me multiple megabytes of content; optimising 20 bytes isn't going to make even the smallest dent if you're trying to pack your site down.
- __jal 8y agoI'm sorry, but that's simply not reflective of reality. There's a reason Yahoo wrote yslow. 20 bytes times billions of requests is absolutely an optimization target; in fact, this is a really easy and low-hanging one at that. Sure, there's also a ton of garbage sites shoving piles of garbage around, but the existence of fast food doesn't mean nice restaurants don't exist.
- LinuxBender 8y agoThey list "date" as being required by protocol. This is not true. The term used in the RFC is "should". It is a nice to have, for additional validation by proxies. In haproxy, you can discard it with: http-response del-header Date
- Rjevski 8y agoJust curious, what would a proxy do with such a header?
- erulabs 8y agoI'd imagine its original design was so that the proxy could choose to honor the response date, rather than just use the current time - technically speaking the date header removes state from the proxy, as the proxy doesn't need to know what time it is to honor cache policies.
- LinuxBender 8y agoProxies used to (and some still do) compare last-modified and date, if the date header is present. [0] They are not required to trust this header as accurate. For reference around and clarification around the Date header, the "should" comes from the loophole that nobody is required to have a time source. The previous RFC's made that harder to understand, as the loophole was in another section. [0] http://devel.squid-cache.org/rproxy/dateheader.html http://devel.squid-cache.org/rproxy/dateheader.html [1] https://tools.ietf.org/html/rfc7231#section-7.1.1.2 https://tools.ietf.org/html/rfc7231#section-7.1.1.2
- rkeene2 8y agoI believe you have mis-interpreted Section 7.1.1.2 of RFC 7231, specifically it is identical to RFC 2616 Section 14.18 in that a Date header MUST be included except for 3 exceptions. They have listed the 3 exceptions first and also the wording that includes "SHOULD" which defines when the origin server should compute the date, but notwithstanding those notes it still notes that the Date header is mandatory for an origin server: "An origin server MUST send a Date header field in all other cases." (where other refers to the 3 exceptions -- HTTP 500-class errors; HTTP 100-class message-less responses; and no-clock systems)
- dewiz 8y agoClient HTTP header I don't want: * referer * user-agent Happy to be wrong, but these shouldn't be mandatory to browse the web, which they kind of are.
- gtirloni 8y agoI too would like a world where all browsers implemented fully the same standard but that's not how it worked out (or it ever works). This is an amusing (scary?) article about the history of the user-agent: https://webaim.org/blog/user-agent-string-history/ https://webaim.org/blog/user-agent-string-history/
- maxk42 8y agoYes but you should be testing browser capabilities not user agents.
- gtirloni 8y agoTotally agree. https://developer.mozilla.org/en-US/docs/Web/HTTP/Browser_detection_using_the_user_agent https://developer.mozilla.org/en-US/docs/Web/HTTP/Browser_de...
- mattmanser 8y agoThe requires more steps and a slower process. User-agent is a one step process. Browser capabilities means returning something back to the browser and potentially coming back to the server. While it has obviously been abused, neither way is ideal. There's no way for a server to say "tell me the browser capabilities before I serve you the request".
- jaytaylor 8y agoNote: I've also written about this on my site with more notes and context: https://jaytaylor.com/writeups/2018/why-referrer-header-empty-nowadays.html https://jaytaylor.com/writeups/2018/why-referrer-header-empt... -- Short version: These days the referrer header rarely makes it through for 2 main classes of reasons [0]. 1. Requests transiting across HTTP <-> HTTPS boundaries do not include the referrer header. 2. The referrer header is frequently disabled by sites (especially search engines and high-traffic sites) through the use a special HTML header meta control tag [1]: <meta name="referrer" content="no-referrer" /> Worry not, though. When client-side Javascript is enabled, ga.js still sends enough information that Google can reconstruct most of everyone's browsing sessions on their backend. Now Google (and only Google) really has all your / our data (generally speaking). :-\ [1] https://stackoverflow.com/questions/6880659/in-what-cases-will-http-referer-be-empty https://stackoverflow.com/questions/6880659/in-what-cases-wi... [0] https://stackoverflow.com/questions/6817595/remove-http-referer https://stackoverflow.com/questions/6817595/remove-http-refe...
- jiveturkey 8y agofairly poor compared to cloudflare blogs
- sqldba 8y agoStep 1: Complain, "Nobody follows the standard." Step 2: Advise, "This is part of the standard but ignore it because it's pointless."
- Steeeve 8y agoI wouldn't trust this entry at all. The author did not do proper research to understand the why's behind the headers that he didn't understand or didn't know well enough.
- yeukhon 8y agoFirst, we should fix user agent. Time to dump that historical baggage.
- ggg9990 8y agoI get that this is data that Fastly has to send but doesn’t get to bill directly to customers, but don’t expect ME to care about this until the average news article stops sending me 10 MB.
- manigandham 8y agoFastly is a CDN that charges by requests + bandwidth, so it absolutely makes money from extra headers on responses no matter how small.
- ggg9990 8y agoI don’t know how they bill. If it is just the Content-Length then they eat the cost of the header.
- ggg9990 8y agoAlso, I can’t see any other reason they’d care. Who does the header harm?
- manigandham 8y agoYou seem to be taking this way too critically. It's a simple article that's looking at the typical headers in responses and showing which ones probably are outdated or unnecessary. If you have 10 hits per day, it doesn't matter. For others that send billions of requests, it might just make a material difference.
- lopmotr 8y agoI got stuck with a website once that was using one of the compression headers - maybe content-encoding to indicate that it's .gz files were gzipped even if the client didn't indicate it supported it. Some browsers would ignore it and just download the file, but others would unzip it. So you got a different file depending on what browser you used! I think wget and chrome behaved differently from each other. I wrote to the site operator who corrected it.
- chrisweekly 8y agoBit of a tangent, but Fastly's CTO gave a terrific talk I attended about a year ago, titled something like "Why load balancing is impossible". My career in consulting has led to a gradual diffusion from my earlier focus on front-end performance optimization, but Fastly retains credibility in my book on a number of fronts.
- _ZeD_ 8y ago>>> Vanity (server, x-powered-by, via) gosh, no. server is no vanity, server is needed to know WHO THE HELL responded you (we are in a very messy cdn selectors + cdns + application layers depending on non obvious rules on (sub)domain and cookies).
- AstralStorm 8y agoThat is supposed to be handled by Host header. Server etc. provides at most redundant debugging info.
- speleding 8y agoThe Host header is in de request, it should not occur in the reply (as the article states)
- randomstring 8y agoWhile working on a HTTP server in 2007 I discovered removing the "Server" header significantly delayed the render time in Firefox. So beware of unexpected side-effects!
- avip 8y agoIn short https://caniuse.com/#feat=contentsecuritypolicy https://caniuse.com/#feat=contentsecuritypolicy
- dijit 8y agoSpeaking of HTTP headers. One I wish more people would use is Accept-Language instead of region/geoip based localization. Practically every site I've come across ignores this header in favour of geoip with the weird and notable exception of Microsoft exchange webmail and Grafana.
- n00bi3s 8y agoThis is a great article!
- deleted 8y ago[deleted]
- itajooba02 8y agoGood article to read, Learned something new today regarding HTTP headers
- dandare 8y agoPlease use a monospaced font when displaying numbers in a table. Otherwise, it is hard to tell which number is 10x bigger than the neighbor.
- ygra 8y agoNo need for a monospace font; tabular numerals should suffice.
- daxterspeed 8y agoI really wish the browser vendors would come together to establish a plan to clean up User-Agent. It's one of the worst offenders in header legacy[1] and fingerprinting. Exposing what browser I am using and it's major version is fine but I don't think every website I visit deserves to know what OS I am using, nor the details of my CPU. [1] https://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/ https://www.nczonline.net/blog/2010/01/12/history-of-the-use... (2010, though little has changed since then).
- gcp 8y agoBrowser vendors can't clean up User-Agent because the websites sniff it and break if it's "wrong" (for any random value of wrong). I'm sure there's a Bugzilla bug about the "X11; Linux x86_64" in the headers, and I'd be terrified to open it.
- torstenvl 8y agoOh God. No. Expires and Pragma are absolutely essential if you're writing a web app to be used by folks stuck behind a walled garden proxy implemented in the dumbest way possible.
- prashnts 8y agoInteresting that their blog itself has the headers they deem unnecessary... Server: Artisanal bits Via: 1.1 varnish,1.1 varnish X-Served-By: cache-sjc3150-SJC, cache-cdg8748-CDG
- dorfsmay 8y ago"no-cache" doesn't prevent caching. "no-store" does. Cache-Control: no-cache, no-store, must-revalidate Mozilla recommends the following to prevent cachine: Cache-Control: no-cache, no-store, must-revalidate" https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ca...