3 ms·
Python is indeed far from ideal for scraping in the modern web, but for only one reason: It can't execute javascript. As a result, js generated content cannot
by ZFCore 5y ago
Python is indeed far from ideal for scraping in the modern web, but for only one reason: It can't execute javascript.
As a result, js generated content cannot be scraped, and python scrapers also get blocked very fast as they don't execute fingerprinting scripts.
- polote 5y agoExecuting javascript and being to render a HTML page are completely different things. To render an HTML page you need a way to create a DOM, donwload all ressources, ... An Node gives you no advantage, as you have to use another lib for that
- traverseda 5y agoDon't execute random javascript from the web in nodejs, nodejs isn't sandboxed the same way that web browsers are and it generally won't work any way. That opens up massive security problems.
- simonw 5y agoTrue - that's why you run scrapers using Playwright or Selenium - both of which can easily be scripted from either JavaScript or Python, while executing website code in a sandboxed browser instance.
- throwawaysea 5y agoIs there a similar guide that walks through step by step how to perform scraping using one of those sandboxes?
- wil421 5y agoNot a guide but a doc I’ve used before is listed below. I use webdriver to open Firefox. https://selenium-python.readthedocs.io/ https://selenium-python.readthedocs.io/
- kristianp 5y agoThat's when you break out PySelenium (if you want to stick with python). Many languages work with selenium drivers, I don't think there's much point in debating which language is best for scraping. Probably one that supports threads, it depends on the scale of course and how much performance you want.