4 ms·
What does this mean for http://www.searchyc.com http://www.searchyc.com? Will it stop too? (Afterall, it must be a bot too, and a robots.txt like this implies t
by eel 17y ago
What does this mean for http://www.searchyc.com http://www.searchyc.com? Will it stop too? (Afterall, it must be a bot too, and a robots.txt like this implies that no bots /should/ scrape the site.) Or will it be allowed?
- qeorge 17y agoThat was my immediate question too. In addition to searchyc, we lose every other HN mash-up present and future. Maybe this will finally bring about an HN API.
- weaksauce 17y agorobots.txt is really only like a note on the front door of your unlocked house that says "please only look at these items in these rooms." or "please do not look in my house" If someone wants to write a program to index the site they can still do so. Google, MSN, Bing, yahoo, et. al. will respect the robots.txt file though.
- mkull 17y agoOr searchyc could simply ignore robots.txt