3 ms·

I'm using lxml with a bit of C for the scrapper behind Printful. My algorithm is way more complex than Readability and I can still parse web pages really fast.

I'm using lxml with a bit of C for the scrapper behind Printful. My algorithm is way more complex than Readability and I can still parse web pages really fast. For example, parsing a long article from NYTimes on a really old machine: fiam@raichu:~/printful$ cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 8 model name : AMD Athlon(tm) XP 2600+ fiam@raichu:~/printful$ python readable.py http://www.nytimes.com/2011/01/25/world/europe/25moscow.html http://www.nytimes.com/2011/01/25/world/europe/25moscow.html > /dev/null Took 0.193097171783 seconds for http://www.nytimes.com/2011/01/25/world/europe/25moscow.html http://www.nytimes.com/2011/01/25/world/europe/25moscow.html I'd say that hosting something like this on App Engine, which prevents you from using C extensions, is not a very good idea. Edit: formatting

Swizec 16y ago
We're using node.js to run readability in. It's actually working out pretty well and is blazing fast on a dedicated computer (or run of the mill PC), the problem is really the overhead involved with virtualization. I think it takes around 0.5 seconds on a PC, a lot of this resulting from a pure javascript implementation of pretty much everything.
- mikeklaas 16y ago
  I found that about 40% of execution time was the the "double br to p's" step (running in mobile safari). Removing that speeds stuff up greatly without compromising quality too much. Also, the publicly available port of readability to python is ridiculously slow... I was about to make it about 10 times faster by re-implementing it myself. I can't release the source code, but the idea is to invert the algorithm: do one pass through the DOM collecting stats about children nodes, text length, and link density, instead of repeated queries at each step. You can get up to 10-20 docs/second on commodity hardware with this approach with an cElementTree DOM and python.

Swizec 16y ago

We're using node.js to run readability in. It's actually working out pretty well and is blazing fast on a dedicated computer (or run of the mill PC), the problem is really the overhead involved with virtualization. I think it takes around 0.5 seconds on a PC, a lot of this resulting from a pure javascript implementation of pretty much everything.

mikeklaas 16y ago
I found that about 40% of execution time was the the "double br to p's" step (running in mobile safari). Removing that speeds stuff up greatly without compromising quality too much. Also, the publicly available port of readability to python is ridiculously slow... I was about to make it about 10 times faster by re-implementing it myself. I can't release the source code, but the idea is to invert the algorithm: do one pass through the DOM collecting stats about children nodes, text length, and link density, instead of repeated queries at each step. You can get up to 10-20 docs/second on commodity hardware with this approach with an cElementTree DOM and python.