4 ms·
What's a good tool/language to write scrapers in these days? A decade ago I was using ruby with mechanize and hpricot. I hope tools have improved since then, es
by andruby 4y ago
What's a good tool/language to write scrapers in these days? A decade ago I was using ruby with mechanize and hpricot. I hope tools have improved since then, especially for scraping sites that use javascript.
- dewey 4y agoReally depends on how big your scraping operation is going to be. These days there's a lot of "managed" providers that give you headless browsers / proxy rotators through an easy API so it's relatively easy to plug them into your code. Examples of these would be https://www.browserless.io https://www.browserless.io or https://www.scrapingbee.com https://www.scrapingbee.com for headless browsers to render JS. From my work experience of working on a large scraping stack with thousands of integrations, I can say that we are very happy with our own custom framework, written in Go (https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery for HTML parsing) and using headless Chrome for JS rendering.
- spfzero 4y agoA fun way, though maybe there are much more productive ways, is to learn Scheme and/or Lisp, and, with a language that has a library for it, convert the html to a big s-expression. Then you have it in a form that is the form of the language itself, where you can literally do anything with it.