3 ms·
Did you do anything to avoid an IP ban for the crawler?
by dreen 11y ago
Did you do anything to avoid an IP ban for the crawler?
- dave5104 11y agoI'm curious about this too, especially since OP says they'd like to continue scraping. I've heard of other people crawling craigslist getting banned pretty fast and even one site getting slapped with a lawsuit [0]. (Which looks like was not entirely successful for craigslist.) Craigslist has even taken some measures to get their users to assign copyright of the content over to craigslist themselves [1]. [0] http://venturebeat.com/2013/05/01/padmapper-craigslist-3taps/ http://venturebeat.com/2013/05/01/padmapper-craigslist-3taps... [1] http://www.on-site.com/what-you-need-to-know-about-craigslist-copyright-assignment/ http://www.on-site.com/what-you-need-to-know-about-craigslis...
- racketracer 11y agoI scraped it probably at least 50 times and nothing has happened. I heard that craigslist only blocks IPs that direct a lot of traffic toward their sites like Padmapper did.