4 ms·
Although we do put the onus on the users to pay attention to robots.txt we realize the reality is that some amount of them won't necessarily respect it. That's
by pranade 13y ago
Although we do put the onus on the users to pay attention to robots.txt we realize the reality is that some amount of them won't necessarily respect it. That's one of the primary reasons behind the way we designed our crawler the way we did -- requiring people to actually specify the links it will visit (as opposed to spidering around sites following all links). Our hope at least is this requires people to put a little thought into the data they want (and where they want it from) before hitting a site.
- adwf 13y agoSounds good enough to me. Might have to keep an eye out for it, hopefully your users will all be good ;) Maybe you could design a feature that gives them a little friendly reminder if they're about to cross someones robots.txt, lets them check that they're aren't about to do something seriously annoying. Sometimes a robots.txt is over zealous and should be crossed, most of the time not though.
- pranade 13y agoAh, yeah that's not a bad idea. Still keep the responsibility on the user, but do our best to keep them honest :)
- loceng 13y agoI believe I remember you previously writing that you're also going to allow site owners to come to login to kimono and turn off spidering for their site?
- pranade 13y agoYes, absolutely... we'll be providing more control for webmasters before we exit beta.