4 ms·
https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt replies Not Found over here
by kauegimenes 7y ago
https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt replies Not Found over here
- mzmcbride 7y agoThis URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related: $ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1 HTTP/2 501 Compare with: $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1 HTTP/2 200
- kauegimenes 7y agoThanks, works from incognito mode in browsers also then :)