3 ms·
Cool site! Curious to know what do you use for scraping, without getting blocked?
by nreece 7y ago
Cool site! Curious to know what do you use for scraping, without getting blocked?
- sphix0r 7y agoThanks a lot! We use scrapy[1]. Our crawler's behavior is very polite so most of them won't block us. Easyjet for example is very strict against crawlers and will block most of them(and us). [1] https://scrapy.org/ https://scrapy.org/
- krn 7y ago> Easyjet for example is very strict against crawlers and will block most of them(and us). It seems that easyJet has outsourced its anti-scraping protection to the company called "Distil Networks"[1]. I wonder how could they block screen scrapers using Chrome on Android with mobile IP addresses. [1] https://www.youtube.com/watch?v=43wh3beHSBo https://www.youtube.com/watch?v=43wh3beHSBo