9 ms·
Their robots.txt file is being delivered as the wrong content type for me. It is being served as a "application/octet-stream" instead of a "text/html" I have n
by UnoriginalGuy 14y ago
Their robots.txt file is being delivered as the wrong content type for me. It is being served as a "application/octet-stream" instead of a "text/html"
I have no idea what consequences this might have since I've never seen another site which does it. Some quick Googling turned up nothing.
Maybe Bing choked on the wrong content type and decided to blackball their entire domain in order to avoid indexing the unindexable?
- X-Istence 14y agoWhy would you want to serve up a robot.txt as "text/html" when in reality it is "text/plain"?