4 ms·
Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get
by markbnj 10y ago
Does this run javascript on the page? I've done quite a bit of scraping with scrapy, and have had to use phantomjs in many cases because static html doesn't get what you're after.
- tekacs 10y agoAt a glance, no - it uses Requests to fetch pages and BeautifulSoup to parse them, the latter of which only parses the HTML into a document object. So static HTML parsing only.
- yes_or_gnome 10y agoI use PhantomJS as well, but (assuming you haven't already) look at CasperJS. It uses PhantomJS, but it is more friendly to use for bigger tasks.
- ddebernardy 10y agoYou could use Splash for JS as well. (Disclaimer: working for Scrapinghub, the main maintainers of Scrapy/Splash.)