4 ms·
This seems unnecessarily complicated and antagonistic. Mostly just to publicly shame a bunch of people. At Reddit we originally blocked a couple of crawlers bu
by jedberg 7y ago
This seems unnecessarily complicated and antagonistic. Mostly just to publicly shame a bunch of people.
At Reddit we originally blocked a couple of crawlers but then realized how pointless that was. The entire robots file[0] is basically now just for google. All of the restrictions are enforced on the server side because there were so many bad bots it didn’t matter if we listed them or not.
[0] https://www.reddit.com/robots.txt https://www.reddit.com/robots.txt
- thinkloop 7y ago> Mostly just to publicly shame a bunch of people. For sure, but that's great, shame can be very effective.
- notafrog 7y agoI was totally expecting /my_shiny_metal_ass to have some kind of easter egg.
- jedberg 7y agoWe used to have one a long time ago when we fronted with nginx and I could just throw a static file up there, but once we switched to everything being dynamic, it wasn't worth the effort to code up a route. :)
- kauegimenes 7y agohttps://www.reddit.com/robots.txt https://www.reddit.com/robots.txt replies Not Found over here
- mzmcbride 7y agoThis URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related: $ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1 HTTP/2 501 Compare with: $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1 HTTP/2 200
- kauegimenes 7y agoThanks, works from incognito mode in browsers also then :)
- wybiral 7y agoGitHub has an interesting entry in theirs [0]. The only user that's in the Disallow section is "/ekansa". He explained it a bit on a Twitter thread [1] where I brought it up a while back and it sounds like they didn't like the fact that he was using GitHub to host XML files for another service because of the traffic from crawlers it created. [0] https://github.com/robots.txt https://github.com/robots.txt [1] https://twitter.com/ekansa/status/1137052076062650368 https://twitter.com/ekansa/status/1137052076062650368
- philpem 7y agoUser-agent: * Allow: /humans.txt Disallow: / Nice!