4 ms·
> I found that their anti-scraping defenses were really difficult to get around What sort of defenses are they using?
by Reedx 8y ago
> I found that their anti-scraping defenses were really difficult to get around
What sort of defenses are they using?
- stevenwoo 8y agoI was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.
- smt88 8y agoThey had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.