cloudlflare has an anti-ddos feature that makes it hard to scrap pages. This feature involves a number of http redirects, and at some stage they need to run jav

cloudlflare has an anti-ddos feature that makes it hard to scrap pages. This feature involves a number of http redirects, and at some stage they need to run javascript in order to get to the next step. The only fix was to scrap these pages was by means of automating the browser with selenium. I was scrapping a big number of web pages for my project here https://github.com/mosermichael/duckduckbang https://github.com/mosermichael/duckduckbang Had to get the description of the sites used by duckduckgo bang operators, so i wrote my own scanning system too...