5 ms·

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, G

by mikece 6y ago

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

diarrhea 6y ago
Don't know about large scales, but just today I threw together a script using selenium, imported pandas to mangle the scraped data and quickly exported to json. For quick and dirty, possibly one-off jobs like that, Python is a great choice.
lemagedurage 6y ago
I think Python makes sense, at least for the prototyping phase. There's a lot of trial and error involved, and Python is quick to write.
freedomben 6y ago
I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.
- isbvhodnvemrwvn 6y ago
  Especially with stuff like Puppeteer which allows you to execute JS in context of the browser (which admittedly can lead to weird bugs as the functions are serialized and lose context)
  - a1sabau 6y ago
    I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content. [1] https://github.com/get-set-fetch/scraper https://github.com/get-set-fetch/scraper
monkeybutton 6y ago
I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.
- RhodesianHunter 6y ago
  I'd say that really depends on your scale and what you're doing with the content you scrape. In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.
  - monkeybutton 6y ago
    Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.
    - RhodesianHunter 6y ago
      My experience was with e-commerce scraping. Not many domains, but a massive catalogue.
    - tluyben2 6y ago
      I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.
jbergstroem 6y ago
The Scrapy library written in Python (https://scrapy.org https://scrapy.org) is excellent for writing and deploying scrapers.

by mikece 6y ago

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

diarrhea 6y ago

Don't know about large scales, but just today I threw together a script using selenium, imported pandas to mangle the scraped data and quickly exported to json. For quick and dirty, possibly one-off jobs like that, Python is a great choice.

lemagedurage 6y ago

I think Python makes sense, at least for the prototyping phase. There's a lot of trial and error involved, and Python is quick to write.

freedomben 6y ago

I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.

isbvhodnvemrwvn 6y ago
Especially with stuff like Puppeteer which allows you to execute JS in context of the browser (which admittedly can lead to weird bugs as the functions are serialized and lose context)
- a1sabau 6y ago
  I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content. [1] https://github.com/get-set-fetch/scraper https://github.com/get-set-fetch/scraper

monkeybutton 6y ago

I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.

RhodesianHunter 6y ago
I'd say that really depends on your scale and what you're doing with the content you scrape. In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.
- monkeybutton 6y ago
  Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.
  - RhodesianHunter 6y ago
    My experience was with e-commerce scraping. Not many domains, but a massive catalogue.
  - tluyben2 6y ago
    I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.

jbergstroem 6y ago

The Scrapy library written in Python (https://scrapy.org https://scrapy.org) is excellent for writing and deploying scrapers.