3 ms·
You don't really get any benefit from writing it in javascript, other than the normal benefits you get from writing anything in javascript. (I say this having v
by traverseda 5y ago
You don't really get any benefit from writing it in javascript, other than the normal benefits you get from writing anything in javascript. (I say this having very little experience with server-side javascript, so take it with a grain of salt)
DoM emulation and selectors are pretty much equivalent between nodejs and python, you can use css or xpath selectors on html/xml content on either of them. Either way you need to emulate something like a DoM, as neither language/execution-environment has a "native" DoM.
You don't want to execute random javascript code from the web inside your scraper, and just being able to parse the scripts doesn't do you much good. So you're not getting the main advantage I think you're suggesting, being able to emulate page javascript, being able to actually run that code.
Generally if you want to interact with javascript you need to do it in another process (I guess a sufficiently advanced sandbox could work too, an interpreter in your interpreter, but so far that doesn't exist). If you're already going to be running that javascript in a different process for security reasons that different process might as well just be a "remote controlled" web browser.
Historically that was done using selenium, which has good python bindings.
Now days it's being done more with playwright, which started out as a nodejs binding but is moving towards python....
Ultimately I think the reason is that there's no real advantage to using javascript and python is a nicer language with a healthier ecosystem, but your mileage my vary.
- cyberge99 5y agoThis is a thoughtful response, I don’t understand why it’s being downvoted.
- traverseda 5y agoMe either, alas
- deleted 5y ago[deleted]
- __app_dev__ 5y agoActually one big advantage I see is the ability to quickly come up with needed functions and code from Browser DevTools then use the exact same code in a node script. Personally I use this method with Puppeteer for advanced pages such as Single Page Apps (SPA) and other pages that depend on JavaScript, CSS, or other features in the page. Another example of an advanced page would be a site where you have to psychical scroll and wait for content to load from a web service. In these cases a headless browser with JavaScript makes the most sense to me. I've found where it gets tricky with JavaScript is if you have a single missing `async/await` you can introduce bugs in your code that take extra time to solve. For simple pages I do like Python and that you don't need `async/await`.
- traverseda 5y agoSelenium and playwright both allow you to inject javascript directly into the page, which can be nice. I see your point though. Also when I do playwright scripting I normally use async/await, so I guess the grass is always greener ;p In python I find a missing async/await is apparently very early on and doesn't really take extra time to solve. Maybe just better tracebacks in python?
- Abishek_Muthian 5y agoIf performance, Especially concurrency matters then we should include Go based libraries in the discussion as well. Colly[1] is a all batteries included scrapping library, But could be bit intimidating for someone new to HTML scrapping. rod [2] is a browser automation library based on DevTools Protocol which adheres to Go'ish way of doing things and so it's very intuitive but comes with overhead of running chromium browser even if its headless. [1] https://github.com/gocolly/colly https://github.com/gocolly/colly [2] https://github.com/go-rod/rod https://github.com/go-rod/rod