4 ms·
Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really diffic
by smt88 8y ago
Craigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around.
They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.
- MegaButts 8y agoI'm guessing that's because of AirBnB and how they got their start by essentially stealing posts from Craigslist.
- massivecali 8y agoI thought Padmapper got their start that way, by essentially providing a better apartment hunt interface than CL had at the time.
- rhizome 8y agoAnd they got the banhammer, too. Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-settle-suit-craigslist-over-use-real-estate-facts https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
- CalRobert 8y agoWhich was a shame, because padmapper stuck it on a map and linked to Craigslist, instead of just taking the content. If Google Maps did it nobody would think twice.
- giancarlostoro 8y agoCraigsList might make them think twice. I don't think they want to overly exhaust their own resources, what they have works for them.
- Reedx 8y ago> I found that their anti-scraping defenses were really difficult to get around What sort of defenses are they using?
- stevenwoo 8y agoI was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.
- smt88 8y agoThey had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.