4 ms·
Access, legality and rate limiting issues come up a lot. We're working on a couple things to address them. The first is an intelligent job distribution system t
by pranade 13y ago
Access, legality and rate limiting issues come up a lot. We're working on a couple things to address them. The first is an intelligent job distribution system that consolidates scrapes across users and hits sites (and pages) at human-like intervals. the second is to create a portal for webmasters that allows them special privileged access to analytics on data being extracted from their sites, and the ability to "turn on or off" kimono APIs if they see fit. this way, via kimono, a webmaster at chumby could "provision" certain kimono users. we're still yet to see whether the later works out. thanks for the input
- Kudos 13y agoUse a user-agent containing a URL to find out who and what you are, and honor my robots.txt. Having a panel for webmasters along with that would be fine.
- pranade 13y agoGreat suggestion... thanks for this one. We're putting this on our list
- thatthatis 13y agoPlease tell me that the robots.txt suggestion is something that you're already doing and the user agent part is whats going on the list.