6 ms·
Very cool.. but what's the implication on the legal side? Many of the sites forbid "automated" scraping of their sites.(sure they can be all circumvented). On t
by sideproject 12y ago
Very cool.. but what's the implication on the legal side? Many of the sites forbid "automated" scraping of their sites.(sure they can be all circumvented). On that note, what about sites like kimonolabs (http://www.kimonolabs.com http://www.kimonolabs.com), they also let you create an API on top of an existing webiste. I imagine it's done by scraping. Would love to hear some thoughts.
- zackboe 12y agoOn the topic of legality/forbidding scraping, Kimono offers webmasters analytics and control of the scraping users do through its service. I'm not sure if this service is actually live, or forthcoming. https://www.kimonolabs.com/webmasters https://www.kimonolabs.com/webmasters
- atmosx 12y agoScraping Amazon might be illegal, or at least not specifically allowed, otherwise there would have been an API. However reading 'robots.txt' I guess that their policy is 'open to scrapping' by search engines. Because what google spiders do to virtually everyone, can be considered web scrapping. [1] http://www.amazon.com/robots.txt http://www.amazon.com/robots.txt