3 ms·

I could not agree more. The combination of decisions seems even weirder. 1. default request header on by default 2. support for deflation off by default WAT?

• mnarayan01 13y ago
  A ridiculous number of sites use that header to determine whether or not they should provide content. Note they don't actually provide encoded content, they just won't provide the non-encoded content if you don't send that header. (I assume this is the reason for it; I have no inside knowledge.)
  • cheald 13y ago
    Can you provide an example of this? I've never run into an instance of this before.
    • avitzurel 13y ago
      Same here, I have never encountered such a case
    • mnarayan01 13y ago
      IIRC http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd did in the past, though it no longer appears to. I know I've run into it in _many_ other places but it's been awhile and I no longer remember the various ones. I should also probably note that they were universally _not_ locations which you would generally be hitting programmatically.
      • cheald 13y ago
        I've been building a webcrawler that casts a pretty wide net and I've yet to run into it. I'm not just talking about API endpoints and stuff - I've never run into this doing general crawls of the web.