4 ms·

What do you use for scraping? I may have a scraping project later this year and would love recommendations.

• djm_ 13y ago
  Scrapy gets a solid recommendation from me. http://scrapy.org/ http://scrapy.org/
• egeozcan 13y ago
  I use custom node.js scripts with these libraries: * request - https://github.com/mikeal/request https://github.com/mikeal/request * async - https://github.com/caolan/async https://github.com/caolan/async * cheerio - https://github.com/cheeriojs/cheerio https://github.com/cheeriojs/cheerio * nedb - https://github.com/louischatriot/nedb https://github.com/louischatriot/nedb
• frabcus 13y ago
  We've got quite an old mailing list full of geeks hand-coding web scrapers, if you want somewhere to ask questions: https://groups.google.com/forum/#!forum/scraperwiki https://groups.google.com/forum/#!forum/scraperwiki
• PuerkitoBio 13y ago
  I've written a couple "polite" crawlers in Go (i.e. obeys robots.txt, delays between requests to the same host). - Fetchbot: https://github.com/PuerkitoBio/fetchbot https://github.com/PuerkitoBio/fetchbot Flexible, similar API to net/http (uses a Handler interface with a simple mux provided, supports middleware, etc.) - gocrawl: https://github.com/PuerkitoBio/gocrawl https://github.com/PuerkitoBio/gocrawl Higher-level, more framework than library. Coupled with goquery (https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery ) to scrape the dom (well, the net/html nodes), this makes custom scrapers trivial to write. (sorry for the self-promoting comment, but this is quite on topic) edit: polite crawlers, not scrapers.