6 ms·

Python web scraping resources

158 points by dchuk 12y ago

rbucks 12y ago
As a self-taught Python and Ruby programmer, I really appreciate this.
jmduke 12y ago
I would highly recommend Scrapy if you plan on doing any serious scraping: http://scrapy.org/ http://scrapy.org/.
- rahimnathwani 12y ago
  This. The Scrapy tutorial is good if you just want to use scrapy to crawl a site and extract a bunch of information, one time. If you want to do scraping as a small part of another Python project, then it can be easier just to use Scrapy's HtmlXPathSelector, which is more forgiving than a real XML parser. import urllib2 from scrapy.selector import HtmlXPathSelector from scrapy.http import TextResponse url = 'http://www.dmoz.org/Computers/Programming/Languages/Python/Books/' my_xpath = '//title/text()' req = urllib2.Request(self.url, headers={'User-Agent' : "Mozilla or whatever"}) url_response = urllib2.urlopen(req) body = url_response.read() response = TextResponse(url = '', body = body, encoding = 'utf-8') hxs = HtmlXPathSelector(response) result = hxs.select(my_xpath).extract()
  - TkTech 12y ago
    HTMLXPathSelector is just a very small wrapper around lxml, it doesn't add anything parsing wise. You might as well just use lxml directly if you don't already have scrapy as a dependency. https://github.com/scrapy/scrapy/tree/master/scrapy/selector https://github.com/scrapy/scrapy/tree/master/scrapy/selector
    - crdoconnor 12y ago
      pyquery's a good alternative too. it's a slightly larger wrapper around lxml that lets you use jquery selectors.
      - klibertp 12y ago
        lxml's `cssselect` method is nice for this - I found that with `xpath` and `cssselect` I have no need for anything else. I use cssselect for simple queries, like "a.something" - which would be needlessly verbose in XPath - and xpath for more complex ones, for example when I need access to axes or I want to apply some simple transform to the data before processing it in Python. Worked very well for me.
    - rahimnathwani 12y ago
      Doh! Too late to edit or delete my original comment :( (And I can't downvote my own comment.)
- crdoconnor 12y ago
  I wouldn't. I used this for a project and then quickly regretted it for the following reasons: * XPath selectors just plain suck. * The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. * Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. * Tons of useless features. Telnet console? wtf? * It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. * For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). IMO this is a classic case of "framework that should have been a library". Here's what I used instead after scrapping scrapy: * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. I'm largely happy with the outcome. The code is less straitjacketed, easier to understand and easier to integrate into other projects if necessary (you don't have the headache of trying to get two frameworks to play together nicely).
  - kmike84 12y ago
    Hey, A good feedback, thanks! > XPath selectors just plain suck. Scrapy supports CSS selectors. > The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. Could you plese give an example? > Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. This is true. But it is a pain to integrate any even-loop based app with another app that is not event-loop based. It is also true that Scrapy is not easy to plug into existing event loop (e.g. if you already have twisted or tornado-based service), but it should be fixed soon. > Tons of useless features. Telnet console? wtf? Telnet console is a Twisted feature; it came almost for free, and it is useful to debug long-running spiders (which can run hours and days). > It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. If you just want to do some post-processing then I don't see why putting code to Scrapy spider is worse than putting it to other script and calling Scrapy from this script. > For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). This may be true.. I don't quite get what is it for :) > IMO this is a classic case of "framework that should have been a library". It can't be a library like requests or mechanize for technical reasons - to make crawling efficient Scrapy uses event loop. It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified. > * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. Celery is also not the easiest piece of software. Scrapy is just a single Python process that doesn't require any databases, etc.; Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.
tomsthumb 12y ago
python selenium bindings are also nice if you absolutely have to deal with information put together by javascript code.
- gee_totes 12y ago
  Have you thought about using a headless web browser like casperjs?
  - Jake232 12y ago
    PhantomJS can be run through Selenium.
    - deleted 12y ago
      [deleted]
- Jake232 12y ago
  I actually cover this (very briefly) here: http://jakeaustwick.me/python-web-scraping-resource/#thesiteisshowingdifferentcontenttomyscraper http://jakeaustwick.me/python-web-scraping-resource/#thesite... I should dedicate a section to it though, will stick that on my to-do list.
  - tomarr 12y ago
    I've always opted for Ghost.py rather than selenium as I've found it uses less memory and pretty capable. Admittedly my scrapes are normally pretty targeted and not over 10k+ sites. One query I had from your piece: "When I've found myself in the unfortunate place of getting my proxies banned before on certain sites, they have been more the happy to switch them out for new IP's for me." If you're getting banned from sites, is it not time to leave them alone? If they don't want your traffic (which the admin/system has judged as too much), should you really be circumventing it? With that and the robots.txt bit (which admittedly you justified), you've got to be careful not to slip into a bit of a grey area with scraping, which people regard suspiciously in the first place.
- 12y ago
halcyondaze 12y ago
I need to scrape an eccomerce site this weekend, and this will be a great resource to keep bookmarked. Thanks.
- miket 12y ago
  Check out http://diffbot.com/products/automatic/product/ http://diffbot.com/products/automatic/product/ to do this fully automated.
  - logn 12y ago
    I'll self-promote too then. https://screenslicer.com https://screenslicer.com
    - BorisMelnik 12y ago
      very cool! probably one of the first web scrapers I've run across with a front end where you can easily show users / investors etc what this does. and it seems to work pretty well. really like your output display as well.
Diastro 12y ago
Little distributed web scraper project I created a while back if anyone is interested / needs resources : https://github.com/Diastro/Zeek https://github.com/Diastro/Zeek
dmritard96 12y ago
A nice write up. At a previous company I built a solution (was forced into java but ultimately the same process...) that used many of these techniques. Some suggested next steps/additional enhancements if you need to do this repeatedly and at scale. Implement global throttling on a per domain basis. Consider some abstraction. I implemented an abstract fetcher (with a number of concrete fetchers that were runtime selectable), and an abstract/concrete parser. Compose a Scraper with these two things. Allow for a runtime switch that will determine which fetcher to use (javascript enabled, straight requests, etc.). If you want to get really fancy, in your database of urls, you can flag urls that need to use a heavier, full fledged browser. For the fetcher, use the selenium bindings. We tested phantomjs and chrome and chrome outperformed phantom. It might have been the java bindings (ghostdriver) but w/e, something you just have to test for yourself. Once we settled on chrome, I built a chrome plugin to block ads and other unrelated calls. This adds LOADs of time. Its pretty tricky but you can inject in a list of well crafted regexes and it drops initial load times dramatically. For the parser, you may want to consider a fallback system. Often times the particular piece of data you want (say a title) can be found in a handful of places on the page. It will make your parsing much more reliable. Compose a 'bot runner' from the Scraper. We had json documents that described fields we were abstracting and all the fallback rules used to locate the needed data. Lastly, the bot runner can be expanding to include things like navigation and other fancier tricks. If you go for broke, build a system for generating bots, think chrome plugin. Don't forget, pruning dead URLs is a tricky little problem but an important one. To scale this whole operation linearly, we used a queue (redis at first, eventually kafka) and storm. Storm allowed us to arbitrary expand and contract our bot runners. Scraping is a problem that just about everyone encounters and alot of the most standard solutions seem to really fall short. Your article is an excellent start.
- meritt 12y ago
  Curious what was the catalyst to switch from Redis to Kafka? Reliability that a given message was received and processed and replaying that to a different consumer in the event of failure?
  - dmritard96 12y ago
benny 12y ago
indepth article! Had to learn most of the stuff the hard way, could need that a couple of weeks ago:) Proxies nowadays are really cheap. Isnt ignoring the robot.txt opening doors for suing? Scraping copyrighted material should be avoided too in my opinion, but i guess that only matters if you get caught:)
chatmasta 12y ago
Really nice. I was just going to comment on how similar this was to scraping I did during my days working in SEO, and then I saw your username! Long time no talk. What's up man? Sounds like we should get in touch.
deleted 12y ago
[deleted]
mushfiq 12y ago
Clean and direct write up. It reminds me when I used to work on crawling/scrapping. Covered most of the topics that needs to know for web crawling.
preinheimer 12y ago
I wish every post/tool out there on scraping covered obeying robots.txt. It's a crap standard, but it's what we've got. "Just ignore it" is a great way to identify yourself as a crappy netizen.
- mdaniel 12y ago
  I'm glad that all of the sites you target want your scraper to access them. The goal in many cases where one would use a scraper is to access information not provided in an API or otherwise encased in HTML. Most of their robots.txt are "User-Agent: *\nDisallow: /\n"
  - preinheimer 12y ago
    Then we have no right to scrape that content. Why is there an implied right to scrape?
victorhooi 12y ago
Very nice, thanks for posting =). Can people suggest any additional resources/reading on scraping/crawling as well? I was hoping to experiment with it in GoLang, but there doesn't seem to be much on crawling/scraping with GoLang, except for GoQuery (https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery)

158 points by dchuk 12y ago

rbucks 12y ago

As a self-taught Python and Ruby programmer, I really appreciate this.

jmduke 12y ago

I would highly recommend Scrapy if you plan on doing any serious scraping: http://scrapy.org/ http://scrapy.org/.

rahimnathwani 12y ago
This. The Scrapy tutorial is good if you just want to use scrapy to crawl a site and extract a bunch of information, one time. If you want to do scraping as a small part of another Python project, then it can be easier just to use Scrapy's HtmlXPathSelector, which is more forgiving than a real XML parser. import urllib2 from scrapy.selector import HtmlXPathSelector from scrapy.http import TextResponse url = 'http://www.dmoz.org/Computers/Programming/Languages/Python/Books/' my_xpath = '//title/text()' req = urllib2.Request(self.url, headers={'User-Agent' : "Mozilla or whatever"}) url_response = urllib2.urlopen(req) body = url_response.read() response = TextResponse(url = '', body = body, encoding = 'utf-8') hxs = HtmlXPathSelector(response) result = hxs.select(my_xpath).extract()
- TkTech 12y ago
  HTMLXPathSelector is just a very small wrapper around lxml, it doesn't add anything parsing wise. You might as well just use lxml directly if you don't already have scrapy as a dependency. https://github.com/scrapy/scrapy/tree/master/scrapy/selector https://github.com/scrapy/scrapy/tree/master/scrapy/selector
  - crdoconnor 12y ago
    pyquery's a good alternative too. it's a slightly larger wrapper around lxml that lets you use jquery selectors.
    - klibertp 12y ago
      lxml's `cssselect` method is nice for this - I found that with `xpath` and `cssselect` I have no need for anything else. I use cssselect for simple queries, like "a.something" - which would be needlessly verbose in XPath - and xpath for more complex ones, for example when I need access to axes or I want to apply some simple transform to the data before processing it in Python. Worked very well for me.
  - rahimnathwani 12y ago
    Doh! Too late to edit or delete my original comment :( (And I can't downvote my own comment.)
crdoconnor 12y ago
I wouldn't. I used this for a project and then quickly regretted it for the following reasons: * XPath selectors just plain suck. * The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. * Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. * Tons of useless features. Telnet console? wtf? * It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. * For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). IMO this is a classic case of "framework that should have been a library". Here's what I used instead after scrapping scrapy: * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. I'm largely happy with the outcome. The code is less straitjacketed, easier to understand and easier to integrate into other projects if necessary (you don't have the headache of trying to get two frameworks to play together nicely).
- kmike84 12y ago
  Hey, A good feedback, thanks! > XPath selectors just plain suck. Scrapy supports CSS selectors. > The item pipeline is way too straitjacketed. You have to do all sorts of fucking around with settings in order to make the sequence of events work in the (programmatic) way you want it because the framework developers 'assumed' there's only one way you'd really want to do it. Could you plese give an example? > Scrapy does not play well with other projects. You can integrate it with django if you want a minimal web UI but it's a pain to do so. This is true. But it is a pain to integrate any even-loop based app with another app that is not event-loop based. It is also true that Scrapy is not easy to plug into existing event loop (e.g. if you already have twisted or tornado-based service), but it should be fixed soon. > Tons of useless features. Telnet console? wtf? Telnet console is a Twisted feature; it came almost for free, and it is useful to debug long-running spiders (which can run hours and days). > It's assumed that the 'end' of the pipeline will be some sort of serialization - to database, xml, json or something. Actually I usually just want to feed into the end of another project without any kind of serialization using plain old python objects. If I want serialization I probably want to do it myself. If you don't want serialization then you want a single process both for crawling and for other tasks. This rules out synchronous solutions - you can't e.g. integrate a crawler with django efficiently without serialization. If you just want to do some post-processing then I don't see why putting code to Scrapy spider is worse than putting it to other script and calling Scrapy from this script. > For some reason DjangoItem didn't really work (although by the time I tried to get it to work I'd kind of given up). This may be true.. I don't quite get what is it for :) > IMO this is a classic case of "framework that should have been a library". It can't be a library like requests or mechanize for technical reasons - to make crawling efficient Scrapy uses event loop. It can (and should) be a library for twisted/tornado/asyncio; it is possible to use Scrapy as a such library now, but this is not straightforward; this should (and will) be simplified. > * mechanize - to mimic a web browser. I used requests sometimes too, but it doesn't really excel at pretending to be a web browser, so for that reason I usually used mechanize as a drop in replacement. * celery - to schedule the crawling / spin off multiple crawlers / rate-limiting / etc. * pyquery - because xpath selectors suck and jquery selectors are better. * python generators - to do pipelining. Celery is also not the easiest piece of software. Scrapy is just a single Python process that doesn't require any databases, etc.; Celery requires to deploy a broker and have a place to store task results; it is also less efficient for IO-bound tasks.

tomsthumb 12y ago

python selenium bindings are also nice if you absolutely have to deal with information put together by javascript code.

gee_totes 12y ago
Have you thought about using a headless web browser like casperjs?
- Jake232 12y ago
  PhantomJS can be run through Selenium.
  - deleted 12y ago
    [deleted]
Jake232 12y ago
I actually cover this (very briefly) here: http://jakeaustwick.me/python-web-scraping-resource/#thesiteisshowingdifferentcontenttomyscraper http://jakeaustwick.me/python-web-scraping-resource/#thesite... I should dedicate a section to it though, will stick that on my to-do list.
- tomarr 12y ago
  I've always opted for Ghost.py rather than selenium as I've found it uses less memory and pretty capable. Admittedly my scrapes are normally pretty targeted and not over 10k+ sites. One query I had from your piece: "When I've found myself in the unfortunate place of getting my proxies banned before on certain sites, they have been more the happy to switch them out for new IP's for me." If you're getting banned from sites, is it not time to leave them alone? If they don't want your traffic (which the admin/system has judged as too much), should you really be circumventing it? With that and the robots.txt bit (which admittedly you justified), you've got to be careful not to slip into a bit of a grey area with scraping, which people regard suspiciously in the first place.
12y ago

halcyondaze 12y ago

I need to scrape an eccomerce site this weekend, and this will be a great resource to keep bookmarked. Thanks.

miket 12y ago
Check out http://diffbot.com/products/automatic/product/ http://diffbot.com/products/automatic/product/ to do this fully automated.
- logn 12y ago
  I'll self-promote too then. https://screenslicer.com https://screenslicer.com
  - BorisMelnik 12y ago
    very cool! probably one of the first web scrapers I've run across with a front end where you can easily show users / investors etc what this does. and it seems to work pretty well. really like your output display as well.

Diastro 12y ago

Little distributed web scraper project I created a while back if anyone is interested / needs resources : https://github.com/Diastro/Zeek https://github.com/Diastro/Zeek

dmritard96 12y ago

A nice write up. At a previous company I built a solution (was forced into java but ultimately the same process...) that used many of these techniques. Some suggested next steps/additional enhancements if you need to do this repeatedly and at scale. Implement global throttling on a per domain basis. Consider some abstraction. I implemented an abstract fetcher (with a number of concrete fetchers that were runtime selectable), and an abstract/concrete parser. Compose a Scraper with these two things. Allow for a runtime switch that will determine which fetcher to use (javascript enabled, straight requests, etc.). If you want to get really fancy, in your database of urls, you can flag urls that need to use a heavier, full fledged browser. For the fetcher, use the selenium bindings. We tested phantomjs and chrome and chrome outperformed phantom. It might have been the java bindings (ghostdriver) but w/e, something you just have to test for yourself. Once we settled on chrome, I built a chrome plugin to block ads and other unrelated calls. This adds LOADs of time. Its pretty tricky but you can inject in a list of well crafted regexes and it drops initial load times dramatically. For the parser, you may want to consider a fallback system. Often times the particular piece of data you want (say a title) can be found in a handful of places on the page. It will make your parsing much more reliable. Compose a 'bot runner' from the Scraper. We had json documents that described fields we were abstracting and all the fallback rules used to locate the needed data. Lastly, the bot runner can be expanding to include things like navigation and other fancier tricks. If you go for broke, build a system for generating bots, think chrome plugin. Don't forget, pruning dead URLs is a tricky little problem but an important one. To scale this whole operation linearly, we used a queue (redis at first, eventually kafka) and storm. Storm allowed us to arbitrary expand and contract our bot runners. Scraping is a problem that just about everyone encounters and alot of the most standard solutions seem to really fall short. Your article is an excellent start.

meritt 12y ago
Curious what was the catalyst to switch from Redis to Kafka? Reliability that a given message was received and processed and replaying that to a different consumer in the event of failure?
- dmritard96 12y ago

benny 12y ago

indepth article! Had to learn most of the stuff the hard way, could need that a couple of weeks ago:) Proxies nowadays are really cheap. Isnt ignoring the robot.txt opening doors for suing? Scraping copyrighted material should be avoided too in my opinion, but i guess that only matters if you get caught:)

chatmasta 12y ago

Really nice. I was just going to comment on how similar this was to scraping I did during my days working in SEO, and then I saw your username! Long time no talk. What's up man? Sounds like we should get in touch.

deleted 12y ago

[deleted]

mushfiq 12y ago

Clean and direct write up. It reminds me when I used to work on crawling/scrapping. Covered most of the topics that needs to know for web crawling.

preinheimer 12y ago

I wish every post/tool out there on scraping covered obeying robots.txt. It's a crap standard, but it's what we've got. "Just ignore it" is a great way to identify yourself as a crappy netizen.

mdaniel 12y ago
I'm glad that all of the sites you target want your scraper to access them. The goal in many cases where one would use a scraper is to access information not provided in an API or otherwise encased in HTML. Most of their robots.txt are "User-Agent: *\nDisallow: /\n"
- preinheimer 12y ago
  Then we have no right to scrape that content. Why is there an implied right to scrape?

victorhooi 12y ago

Very nice, thanks for posting =). Can people suggest any additional resources/reading on scraping/crawling as well? I was hoping to experiment with it in GoLang, but there doesn't seem to be much on crawling/scraping with GoLang, except for GoQuery (https://github.com/PuerkitoBio/goquery https://github.com/PuerkitoBio/goquery)