4 ms·
To me it's mainly the following three reasons, but take it with a grain of salt since my JS is not as fluent as Python. 1. the async nature of JS is surprising
by thrdbndndn 3y ago
To me it's mainly the following three reasons, but take it with a grain of salt since my JS is not as fluent as Python.
1. the async nature of JS is surprisingly detrimental when writing scraping script. It's hard to describe, but it makes have a mental image of the whole code base or workflow harder. Writing mostly sync code and only use things like ThreadPoolExecutor (not even Threading directly) when necessary has been much easier for me to write clean, easy-to-maintain code.
2. I really don't like the syntax of loops or iterations in JS, and there are a lot of them in web scraping.
3. String processing and/or data re-shaping feels harder in JS. The built-in functions often feel unintuitive.
- Chiron1991 3y agoI agree with anything you said, and: 4. Having the scraped data in Python-land makes it sometimes way easier to dump it into an analysis landscape, which is probably Python, too.
- giantrobot 3y ago> String processing and/or data re-shaping feels harder in JS. The built-in functions often feel unintuitive. Hey don't worry there's probably a library that does it for you! It only pulls down a half gigabyte of dependencies to left-pad strings! I hate the JavaScript ecosystem so fucking much.
- spaniard89277 3y agoI've had some experiences with selenium and now I'm using puppeteer, and I honestly don't see the problem with JS. It's true that I have not much experience coding but it seems to me that Pupeteer + Flask serving ML to extract data is the cake. Also, being able to play around evaluating expressions in pupeteer, etc, makes it manageable. Maybe I lack experience but I don't see JS being a barrier. I would like to know what kind of string work are you doing. I can't imagine being dependent on parsing strings and such, that looks very easy to break, even easier that css selector dance.
- deanishe 3y agoString processing in general is just easier in Python. As a basic example, `lstrip()` and `rstrip()` trim whitespace by default, but can also remove any characters you'd like from the end(s) of your string. You'd need to code that up yourself in JS. Happens quite a lot.