4 ms·
A ridiculous number of sites use that header to determine whether or not they should provide content. Note they don't actually provide encoded content, they jus
by mnarayan01 13y ago
A ridiculous number of sites use that header to determine whether or not they should provide content. Note they don't actually provide encoded content, they just won't provide the non-encoded content if you don't send that header. (I assume this is the reason for it; I have no inside knowledge.)
- cheald 13y agoCan you provide an example of this? I've never run into an instance of this before.
- avitzurel 13y agoSame here, I have never encountered such a case
- mnarayan01 13y agoIIRC http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd did in the past, though it no longer appears to. I know I've run into it in _many_ other places but it's been awhile and I no longer remember the various ones. I should also probably note that they were universally _not_ locations which you would generally be hitting programmatically.
- cheald 13y agoI've been building a webcrawler that casts a pretty wide net and I've yet to run into it. I'm not just talking about API endpoints and stuff - I've never run into this doing general crawls of the web.