3 ms·

There are numerous web-scraping tools out there, many are easy enough to use for non-programmers (Connotate and Mozenda for instance) so, while you have a point

by meritt 13y ago

There are numerous web-scraping tools out there, many are easy enough to use for non-programmers (Connotate and Mozenda for instance) so, while you have a point, precedent shows that web scraping is prominent and not going away anytime soon. I applaud innovation in this area. Your #2 is simply not plausible unless Kimono carefully follows robots.txt and consistent user-agent, which is rarely the case for a competent web-scraping platform. Legality is also a strong word. Web scraping is rarely illegal in the sense that you're committing a crime. A company could definitely issue a C&D and potentially go after you in a civil lawsuit but it's very infrequent (sadly not so much anymore with all the bullshit CFAA suits against "hackers" lately) that a government entity would pursue someone for scraping data. The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping.

mbesto 13y ago
Precedent is also a strong word. There are numerous people on the road who go over the speed limit. It's prominent and not going away anytime soon. This is not a cause for precedence. Precedence implies that a legal case take place in order to dictate the law. AFAIK, between your unsubstantiated comment and my general knowledge of this area, the reason no one has provided a scraper service at scale isn't because the technology or execution wasn't right, but rather that it's simply illegal to do so. > The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping. That's simply not true. Here's an example: http://www.yelp.com/static?country_=US&p=tos http://www.yelp.com/static?country_=US&p=tos "By accessing or using the Site, you are agreeing to these Terms and concluding a legally binding contract with Yelp Inc., a Delaware corporation headquartered in San Francisco, California ("Yelp"). Do not access or use the Site if you are unwilling or unable to be bound by the Terms." "Use any robot, spider, site search/retrieval application, or other automated device, process or means to access, retrieve, scrape, or index any portion of the Site or any Site Content;" My overall point? Kimonolabs is slowly painting a target on their back and it's only a matter of time before they'll have to "lawyer up".

by meritt 13y ago

mbesto 13y ago

Precedent is also a strong word. There are numerous people on the road who go over the speed limit. It's prominent and not going away anytime soon. This is not a cause for precedence. Precedence implies that a legal case take place in order to dictate the law. AFAIK, between your unsubstantiated comment and my general knowledge of this area, the reason no one has provided a scraper service at scale isn't because the technology or execution wasn't right, but rather that it's simply illegal to do so. > The general consensus is if the information is available to end users with a regular web browser, no logins or agreement checkboxes, then it's fair game for web scraping. That's simply not true. Here's an example: http://www.yelp.com/static?country_=US&p=tos http://www.yelp.com/static?country_=US&p=tos "By accessing or using the Site, you are agreeing to these Terms and concluding a legally binding contract with Yelp Inc., a Delaware corporation headquartered in San Francisco, California ("Yelp"). Do not access or use the Site if you are unwilling or unable to be bound by the Terms." "Use any robot, spider, site search/retrieval application, or other automated device, process or means to access, retrieve, scrape, or index any portion of the Site or any Site Content;" My overall point? Kimonolabs is slowly painting a target on their back and it's only a matter of time before they'll have to "lawyer up".