4 ms·

Thanks a lot! We use scrapy[1]. Our crawler's behavior is very polite so most of them won't block us. Easyjet for example is very strict against crawlers and wi

Thanks a lot! We use scrapy[1]. Our crawler's behavior is very polite so most of them won't block us. Easyjet for example is very strict against crawlers and will block most of them(and us). [1] https://scrapy.org/ https://scrapy.org/

krn 7y ago
> Easyjet for example is very strict against crawlers and will block most of them(and us). It seems that easyJet has outsourced its anti-scraping protection to the company called "Distil Networks"[1]. I wonder how could they block screen scrapers using Chrome on Android with mobile IP addresses. [1] https://www.youtube.com/watch?v=43wh3beHSBo https://www.youtube.com/watch?v=43wh3beHSBo