3 ms·
I'm sure it's technically possible to scrape Amazon. I'm much less confident that it would be legal.
by eli 12y ago
I'm sure it's technically possible to scrape Amazon. I'm much less confident that it would be legal.
- mooreds 12y agoThis is the key point. If all you are doing is scraping prices for a personal project, questionable legality is ok (even if clear legality would be preferable). But if you are trying to build a business, ya gotta stay on the right side of the law.
- toomuchtodo 12y agoWhat law prevents you from scrapping publicly available site?
- minouye 12y agoSupposedly the Computer Fraud and Abuse Act. http://en.wikipedia.org/wiki/Craigslist_v._3Taps http://en.wikipedia.org/wiki/Craigslist_v._3Taps
- eli 12y agohttp://en.wikipedia.org/wiki/Web_scraping#Legal_issues http://en.wikipedia.org/wiki/Web_scraping#Legal_issues I don't think it's mentioned on the wiki but I think there might be a DMCA Anti-Circumvention case too if the site you're scraping has technical measures to prevent data from being copied.
- wrath 12y agoAmazon has made it much harder to scrape their site in recent months. They now give captchas to bots that try and access their site to often by the same IP address. Unless you have a very large pool of low cost proxies, crawling millions of prices on a daily basis is challenging. I think this is what he's referring to when he says """and has lately made it very difficult to get the data in other ways."""
- toomuchtodo 12y agoThe best way to do this is with your users using a Chrome extension to feed the data back to you. User lands on a page? Send the data back to your scrapper processing API.
- jarkarj 12y agoI've been scraping Amazon for the past few years. Yeah the captcha that they introduced around last February is a pain, but there are ways around it. I have and still do make my living from scraping (not just Amazon) - lemme know if you are interested and maybe I can help someone out.