for mostly static pages requests/pycurl + beautifulsoup more than sufficient. For advance scraping, take a look at scrapy. for javascript heavy pages most peop

for mostly static pages requests/pycurl + beautifulsoup more than sufficient. For advance scraping, take a look at scrapy. for javascript heavy pages most people rely on selenium webdriver. However you can also try hlspy (https://github.com/kanishka-linux/hlspy https://github.com/kanishka-linux/hlspy), which is a little utility I made a while ago for dealing with javascript heavy pages for simple usage.