3 ms·
What Google Won't Let MS Click
- chengmi 19y agoDoes this even matter? How many bots actually respect robots.txt?
- gojomo 19y agoMisleading headline: the disallows affect all crawlers equally. Missing context: what is someone, expert at reading robots.txt or not, expected to find interesting in the target link? Bad submission.
- Goladus 19y agorobots.txt is not a security feature. It's a courtesy that saves resources and makes life a lot easier for everyone who is willing to play by the rules.
- rainsill 19y agoThis is a bad submission. False headline. How do you vote an article down?
- rms 19y agoMaking a comment saying it is a bad submission is great, just please don't say "This article should be on reddit!!!!!!!!"