5 ms·

I would highly recommend Scrapy if you plan on doing any serious scraping: http://scrapy.org/ http://scrapy.org/.

by jmduke 12y ago

rahimnathwani 12y ago
This. The Scrapy tutorial is good if you just want to use scrapy to crawl a site and extract a bunch of information, one time. If you want to do scraping as a small part of another Python project, then it can be easier just to use Scrapy's HtmlXPathSelector, which is more forgiving than a real XML parser. import urllib2 from scrapy.selector import HtmlXPathSelector from scrapy.http import TextResponse url = 'http://www.dmoz.org/Computers/Programming/Languages/Python/Books/' my_xpath = '//title/text()' req = urllib2.Request(self.url, headers={'User-Agent' : "Mozilla or whatever"}) url_response = urllib2.urlopen(req) body = url_response.read() response = TextResponse(url = '', body = body, encoding = 'utf-8') hxs = HtmlXPathSelector(response) result = hxs.select(my_xpath).extract()
- TkTech 12y ago
  HTMLXPathSelector is just a very small wrapper around lxml, it doesn't add anything parsing wise. You might as well just use lxml directly if you don't already have scrapy as a dependency. https://github.com/scrapy/scrapy/tree/master/scrapy/selector https://github.com/scrapy/scrapy/tree/master/scrapy/selector
  - crdoconnor 12y ago
    pyquery's a good alternative too. it's a slightly larger wrapper around lxml that lets you use jquery selectors.
    - klibertp 12y ago
      lxml's `cssselect` method is nice for this - I found that with `xpath` and `cssselect` I have no need for anything else. I use cssselect for simple queries, like "a.something" - which would be needlessly verbose in XPath - and xpath for more complex ones, for example when I need access to axes or I want to apply some simple transform to the data before processing it in Python. Worked very well for me.
  - rahimnathwani 12y ago
    Doh! Too late to edit or delete my original comment :( (And I can't downvote my own comment.)
crdoconnor 12y ago
I wouldn't. I used this for a project and then quickly regretted it for the following reasons: * XPath selectors just plain suck. * The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. * Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. * Tons of useless features. Telnet console? wtf? * It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. * For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). IMO this is a classic case of "framework that should have been a library". Here's what I used instead after scrapping scrapy: * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. I'm largely happy with the outcome. The code is less straitjacketed, easier to understand and easier to integrate into other projects if necessary (you don't have the headache of trying to get two frameworks to play together nicely).
- kmike84 12y ago
  Hey, A good feedback, thanks! > XPath selectors just plain suck. Scrapy supports CSS selectors. > The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. Could you plese give an example? > Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. This is true. But it is a pain to integrate any even-loop based app with another app that is not event-loop based. It is also true that Scrapy is not easy to plug into existing event loop (e.g. if you already have twisted or tornado-based service), but it should be fixed soon. > Tons of useless features. Telnet console? wtf? Telnet console is a Twisted feature; it came almost for free, and it is useful to debug long-running spiders (which can run hours and days). > It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. If you just want to do some post-processing then I don't see why putting code to Scrapy spider is worse than putting it to other script and calling Scrapy from this script. > For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). This may be true.. I don't quite get what is it for :) > IMO this is a classic case of "framework that should have been a library". It can't be a library like requests or mechanize for technical reasons - to make crawling efficient Scrapy uses event loop. It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified. > * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. Celery is also not the easiest piece of software. Scrapy is just a single Python process that doesn't require any databases, etc.; Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.

I would highly recommend Scrapy if you plan on doing any serious scraping: http://scrapy.org/ http://scrapy.org/.

by jmduke 12y ago

I would highly recommend Scrapy if you plan on doing any serious scraping: http://scrapy.org/ http://scrapy.org/.

This. The Scrapy tutorial is good if you just want to use scrapy to crawl a site and extract a bunch of information, one time. If you want to do scraping as a small part of another Python project, then it can be easier just to use Scrapy's HtmlXPathSelector, which is more forgiving than a real XML parser. import urllib2 from scrapy.selector import HtmlXPathSelector from scrapy.http import TextResponse url = 'http://www.dmoz.org/Computers/Programming/Languages/Python/Books/' my_xpath = '//title/text()' req = urllib2.Request(self.url, headers={'User-Agent' : "Mozilla or whatever"}) url_response = urllib2.urlopen(req) body = url_response.read() response = TextResponse(url = '', body = body, encoding = 'utf-8') hxs = HtmlXPathSelector(response) result = hxs.select(my_xpath).extract()

TkTech 12y ago
HTMLXPathSelector is just a very small wrapper around lxml, it doesn't add anything parsing wise. You might as well just use lxml directly if you don't already have scrapy as a dependency. https://github.com/scrapy/scrapy/tree/master/scrapy/selector https://github.com/scrapy/scrapy/tree/master/scrapy/selector
- crdoconnor 12y ago
  pyquery's a good alternative too. it's a slightly larger wrapper around lxml that lets you use jquery selectors.
  - klibertp 12y ago
    lxml's `cssselect` method is nice for this - I found that with `xpath` and `cssselect` I have no need for anything else. I use cssselect for simple queries, like "a.something" - which would be needlessly verbose in XPath - and xpath for more complex ones, for example when I need access to axes or I want to apply some simple transform to the data before processing it in Python. Worked very well for me.
- rahimnathwani 12y ago
  Doh! Too late to edit or delete my original comment :( (And I can't downvote my own comment.)

crdoconnor 12y ago

I wouldn't. I used this for a project and then quickly regretted it for the following reasons: * XPath selectors just plain suck. * The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. * Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. * Tons of useless features. Telnet console? wtf? * It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. * For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). IMO this is a classic case of "framework that should have been a library". Here's what I used instead after scrapping scrapy: * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. I'm largely happy with the outcome. The code is less straitjacketed, easier to understand and easier to integrate into other projects if necessary (you don't have the headache of trying to get two frameworks to play together nicely).

kmike84 12y ago
Hey, A good feedback, thanks! > XPath selectors just plain suck. Scrapy supports CSS selectors. > The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. Could you plese give an example? > Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. This is true. But it is a pain to integrate any even-loop based app with another app that is not event-loop based. It is also true that Scrapy is not easy to plug into existing event loop (e.g. if you already have twisted or tornado-based service), but it should be fixed soon. > Tons of useless features. Telnet console? wtf? Telnet console is a Twisted feature; it came almost for free, and it is useful to debug long-running spiders (which can run hours and days). > It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. If you just want to do some post-processing then I don't see why putting code to Scrapy spider is worse than putting it to other script and calling Scrapy from this script. > For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). This may be true.. I don't quite get what is it for :) > IMO this is a classic case of "framework that should have been a library". It can't be a library like requests or mechanize for technical reasons - to make crawling efficient Scrapy uses event loop. It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified. > * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. Celery is also not the easiest piece of software. Scrapy is just a single Python process that doesn't require any databases, etc.; Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.