4 ms·

Fast scraping in Python with Asyncio

96 points by watermel0n 13y ago

drakaal 13y ago
Looks ok. I agree with author many people roll their own. Depending what you want to scrape, this might work better for you. https://www.mashape.com/stremor/stremor-content-extractor-fast-web-page-extraction-for-simple-and-magazine-templates#!documentation https://www.mashape.com/stremor/stremor-content-extractor-fa... It will scrape articles even if they are in multiple divs. It even works on magazine layouts like TheVerge.com that doesn't work in Readability (py, api, or js).
bfwi 13y ago
You can also use grequests (originally a part of requests). It has a grequests.map() function, that I use exactly for this scenario.
- madjar 13y ago
  I used that at some point, but I tend to prefer the explicit yield of asyncio to the implicit context change of greenlets. (This blog post is a good explanation of that: https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html)
avitzurel 13y ago
Code samples on your blog are really hard to read. (the syntax highlighting colors make it impossible)
- mixmastamyk 13y ago
  I brought down the white background to #aaa and it made the code much easier to read.
thomasahle 13y ago
It's very interesting to see the Python libraries starting to experiment with the power of the new 'yield from' statement.
- fafhrd91 13y ago
  it becomes more interesting when you start read streaming server request and then pipe into streaming client request. `yield from` and asyncio makes everything much more simple.
bcl 13y ago
Here are a number of asyncio examples, including Guido's web crawler - https://code.google.com/p/tulip/source/browse/examples/ https://code.google.com/p/tulip/source/browse/examples/
kclay 13y ago
I tend to use scrapy when doing python scraping but its a massive setup, this will be good for smaller projects.
amckenna 13y ago
This looks very similar to a Python library called Blackmamba I have been using on Linux for async requests. If you need support down to Python 2.6 and similar functionality it's worth checking out: http://rootfoo.org/blackmamba http://rootfoo.org/blackmamba https://github.com/rootfoo/blackmamba https://github.com/rootfoo/blackmamba
rdtsc 13y ago
Even better use gevent or eventlet. Then don't have to worry about @asyncio.coroutine, yielding, vs yielding from and do so on.
- jonathan_s 13y ago
  Like the article of @madjar in the comment above explains. There is a reason for definitely not using gevent.
  - rdtsc 13y ago
    I read that article and don't agree. Those methods (gevent and eventlet) are the default and typical way of building concurrent python services. That article conflates threads, shared state and explicit IO awareness and dispatching. Those can be different in each case. Python doesn't have isolated heaps like Erlang but it is still possible to have lightweight threads with a queue and not share by convention data between lightweight threads. Having or not having @coroutines or yields or deferreds or inlineDeferreds doesn't make a difference in that case. In general, a set of jumping callback and errback functions are a lot worse concurrency mechanism than threads. A callback chain is a poor man's threads, it is just spread non-locally across the whole code without a way to properly synchronize. Threads done right, should make it easy to organize data locally. This is quite the opposite of what that article claims. A set of spread around callback functions, do the opposite. Whether yields or inlineDeferreds are used are almost orthogonal to the above. All they do is help you localize data (which callbacks don't) but now you have to explicitly care where and when you do IO. You might not even care in real life but they force you to. That is stupid. Oh and library fragmentation. There are now 3+concurrency frameworks for Python. They all look different. And because of this bubbling of IO primitives up to the surface of the API, make it almost impossible to share libraries between them.
preinheimer 13y ago
I'd really like it if articles on scraping included robots.txt support.
- nashequilibrium 13y ago
  rp = robotparser.RobotFileParser() rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt") rp.read() # Reads the robots.txt rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news') >>>> True
- NoodleIncident 13y ago
  The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine. Copypasta of dead comment: nashequilibrium 13 hours ago | link [dead] rp = robotparser.RobotFileParser() rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt") rp.read() # Reads the robots.txt rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news') >>>> True
deleted 13y ago
[deleted]
jessaustin 13y ago
s/scrapper/scraper/
riobard 13y ago
I'm not sure if it's just me, but after years of writing Python code with various async I/O libs and then used Go for a while, it felt so much better when the language/runtime handles async I/O for me and I do not have to think about event loops at all. Python used to be the clean and simple language for me, but all those ”hacks” to get async right just makes the final code ugly. The cleanness of Go wins me over instantly. Even though the two languages are not directly comparable, the foundations of Go seems to me much better and solid. It reminds me of Rob Pike's essay “Less is exponentially more” http://commandcenter.blogspot.ca/2012/06/less-is-exponentially-more.html http://commandcenter.blogspot.ca/2012/06/less-is-exponential...
- natmaster 13y ago
  Try gevent or eventlet. Real pythonistas use these, instead verbose java-esque coroutines.
- coldtea 13y ago
  To paraphrase JWZ, "Python is big because your needs are big". I don't see anything "hacky" to get async right. It's a simple library. It's just not first class (then again, no reason it should be, isn't it not first class in Rust too?).
bwood 13y ago
Clearly this would be faster than using the requests library to scrape pages in serial, but would it also be faster than using the requests library in several threads at once? Besides the simplified syntax, what is the advantage of asyncio over a threaded scraper?
- madjar 13y ago
  I would say the performance are equivalent, with maybe some overhead if there is some heavy thread spawning. I'll run some benchmarks at some point. One of the biggest advantages is the fact this is not multi-threading, so context switches can only happen at precise points (yield from). This makes it easier to reason about the code.

96 points by watermel0n 13y ago

Looks ok. I agree with author many people roll their own. Depending what you want to scrape, this might work better for you. https://www.mashape.com/stremor/stremor-content-extractor-fast-web-page-extraction-for-simple-and-magazine-templates#!documentation https://www.mashape.com/stremor/stremor-content-extractor-fa... It will scrape articles even if they are in multiple divs. It even works on magazine layouts like TheVerge.com that doesn't work in Readability (py, api, or js).

bfwi 13y ago

You can also use grequests (originally a part of requests). It has a grequests.map() function, that I use exactly for this scenario.

madjar 13y ago
I used that at some point, but I tend to prefer the explicit yield of asyncio to the implicit context change of greenlets. (This blog post is a good explanation of that: https://glyph.twistedmatrix.com/2014/02/unyielding.html https://glyph.twistedmatrix.com/2014/02/unyielding.html)

avitzurel 13y ago

Code samples on your blog are really hard to read. (the syntax highlighting colors make it impossible)

mixmastamyk 13y ago
I brought down the white background to #aaa and it made the code much easier to read.

thomasahle 13y ago

It's very interesting to see the Python libraries starting to experiment with the power of the new 'yield from' statement.

fafhrd91 13y ago
it becomes more interesting when you start read streaming server request and then pipe into streaming client request. `yield from` and asyncio makes everything much more simple.

bcl 13y ago

Here are a number of asyncio examples, including Guido's web crawler - https://code.google.com/p/tulip/source/browse/examples/ https://code.google.com/p/tulip/source/browse/examples/

kclay 13y ago

I tend to use scrapy when doing python scraping but its a massive setup, this will be good for smaller projects.

amckenna 13y ago

This looks very similar to a Python library called Blackmamba I have been using on Linux for async requests. If you need support down to Python 2.6 and similar functionality it's worth checking out: http://rootfoo.org/blackmamba http://rootfoo.org/blackmamba https://github.com/rootfoo/blackmamba https://github.com/rootfoo/blackmamba

rdtsc 13y ago

Even better use gevent or eventlet. Then don't have to worry about @asyncio.coroutine, yielding, vs yielding from and do so on.

jonathan_s 13y ago
Like the article of @madjar in the comment above explains. There is a reason for definitely not using gevent.
- rdtsc 13y ago
  I read that article and don't agree. Those methods (gevent and eventlet) are the default and typical way of building concurrent python services. That article conflates threads, shared state and explicit IO awareness and dispatching. Those can be different in each case. Python doesn't have isolated heaps like Erlang but it is still possible to have lightweight threads with a queue and not share by convention data between lightweight threads. Having or not having @coroutines or yields or deferreds or inlineDeferreds doesn't make a difference in that case. In general, a set of jumping callback and errback functions are a lot worse concurrency mechanism than threads. A callback chain is a poor man's threads, it is just spread non-locally across the whole code without a way to properly synchronize. Threads done right, should make it easy to organize data locally. This is quite the opposite of what that article claims. A set of spread around callback functions, do the opposite. Whether yields or inlineDeferreds are used are almost orthogonal to the above. All they do is help you localize data (which callbacks don't) but now you have to explicitly care where and when you do IO. You might not even care in real life but they force you to. That is stupid. Oh and library fragmentation. There are now 3+concurrency frameworks for Python. They all look different. And because of this bubbling of IO primitives up to the surface of the API, make it almost impossible to share libraries between them.

preinheimer 13y ago

I'd really like it if articles on scraping included robots.txt support.

nashequilibrium 13y ago
rp = robotparser.RobotFileParser() rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt") rp.read() # Reads the robots.txt rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news') >>>> True
NoodleIncident 13y ago
The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine. Copypasta of dead comment: nashequilibrium 13 hours ago | link [dead] rp = robotparser.RobotFileParser() rp.set_url("https://news.ycombinator.com/news/robots.txt" https://news.ycombinator.com/news/robots.txt") rp.read() # Reads the robots.txt rp.can_fetch("*", 'https://news.ycombinator.com/news' https://news.ycombinator.com/news') >>>> True

deleted 13y ago

[deleted]

jessaustin 13y ago

s/scrapper/scraper/

riobard 13y ago

I'm not sure if it's just me, but after years of writing Python code with various async I/O libs and then used Go for a while, it felt so much better when the language/runtime handles async I/O for me and I do not have to think about event loops at all. Python used to be the clean and simple language for me, but all those ”hacks” to get async right just makes the final code ugly. The cleanness of Go wins me over instantly. Even though the two languages are not directly comparable, the foundations of Go seems to me much better and solid. It reminds me of Rob Pike's essay “Less is exponentially more” http://commandcenter.blogspot.ca/2012/06/less-is-exponentially-more.html http://commandcenter.blogspot.ca/2012/06/less-is-exponential...

natmaster 13y ago
Try gevent or eventlet. Real pythonistas use these, instead verbose java-esque coroutines.
coldtea 13y ago
To paraphrase JWZ, "Python is big because your needs are big". I don't see anything "hacky" to get async right. It's a simple library. It's just not first class (then again, no reason it should be, isn't it not first class in Rust too?).

bwood 13y ago

Clearly this would be faster than using the requests library to scrape pages in serial, but would it also be faster than using the requests library in several threads at once? Besides the simplified syntax, what is the advantage of asyncio over a threaded scraper?

madjar 13y ago
I would say the performance are equivalent, with maybe some overhead if there is some heavy thread spawning. I'll run some benchmarks at some point. One of the biggest advantages is the fact this is not multi-threading, so context switches can only happen at precise points (yield from). This makes it easier to reason about the code.