4 ms·

Would you please share some code samples or favorite blog posts on how you're using phpQuery as part of a scrape app? What set of tools/libraries are you using

by Sukotto 16y ago

Would you please share some code samples or favorite blog posts on how you're using phpQuery as part of a scrape app? What set of tools/libraries are you using? I'd really like to hear more about the current state of the art (without you telling any company secrets). I have quite a lot of experience scraping utility and government websites (no javascript) in perl + LWP... but I'm getting a little tired of perl and am looking to give a new toolset a try. Perferably one that can handle a broader range of modern websites.

asnyder 16y ago
php|architect's recently released book entirely on this topic: http://www.phparch.com/books/phparchitects-guide-to-web-scraping-with-php/ http://www.phparch.com/books/phparchitects-guide-to-web-scra...
l4u 16y ago
I have been using spynner, written in python. It uses pyQtWebkit, and jQuery. Use the trunk version at http://code.google.com/p/spynner/ http://code.google.com/p/spynner/
justliving 16y ago
I have been using quite a bit of python and beautifulsoup. http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/
encoderer 16y ago
I will throw you 1 bone, but past that, I'm careful because my work here is not my own, it's my employers. libxml_use_internal_errors(true); This little function call is the secret to solving what seem at first to be intractable memory-leaks. The trouble is the scraper uses libxml, and libxml issues a notice/warning every time the HTML is malformed. Without this call, those errs will bubble up to the PHP error handler and it'll murder performance and memory usage. One more i suppose... If you scrape inside a loop (and unless you're using a distributed job queue,if you're scraping more than one URL at a time you almost certainly are), missing a unloadDocument() call is going to cost you each time you iterate. The objects it creates, IIRC, have some circular dependency issues and if you don't explicitly unloadDocument you'll run into trouble. (Suppose it should be OK tho if you've enabled that GC feature in PHP 5.3) And, generally, a tip... sometimes it's tempting to write a simple regex instead of a chain like, say, pq($this->node->find('a')->get(0))->attr('href') Write the chain. Regex is just too brittle.
- Sukotto 16y ago
  Thanks for the tips. You most likely saved me a bunch of headache and I appreciate it. I'm pretty intrigued by a library that can apparently handle ajax/json updates and content creation. Heh... I thought PHP was only for page generation and had no idea you could purpose it for something like webscraping. So it should be fun playing with it. My email's in my profile if you ever want to talk shop.

by Sukotto 16y ago

asnyder 16y ago

php|architect's recently released book entirely on this topic: http://www.phparch.com/books/phparchitects-guide-to-web-scraping-with-php/ http://www.phparch.com/books/phparchitects-guide-to-web-scra...

l4u 16y ago

I have been using spynner, written in python. It uses pyQtWebkit, and jQuery. Use the trunk version at http://code.google.com/p/spynner/ http://code.google.com/p/spynner/

justliving 16y ago

I have been using quite a bit of python and beautifulsoup. http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/

encoderer 16y ago

I will throw you 1 bone, but past that, I'm careful because my work here is not my own, it's my employers. libxml_use_internal_errors(true); This little function call is the secret to solving what seem at first to be intractable memory-leaks. The trouble is the scraper uses libxml, and libxml issues a notice/warning every time the HTML is malformed. Without this call, those errs will bubble up to the PHP error handler and it'll murder performance and memory usage. One more i suppose... If you scrape inside a loop (and unless you're using a distributed job queue,if you're scraping more than one URL at a time you almost certainly are), missing a unloadDocument() call is going to cost you each time you iterate. The objects it creates, IIRC, have some circular dependency issues and if you don't explicitly unloadDocument you'll run into trouble. (Suppose it should be OK tho if you've enabled that GC feature in PHP 5.3) And, generally, a tip... sometimes it's tempting to write a simple regex instead of a chain like, say, pq($this->node->find('a')->get(0))->attr('href') Write the chain. Regex is just too brittle.

Sukotto 16y ago
Thanks for the tips. You most likely saved me a bunch of headache and I appreciate it. I'm pretty intrigued by a library that can apparently handle ajax/json updates and content creation. Heh... I thought PHP was only for page generation and had no idea you could purpose it for something like webscraping. So it should be fun playing with it. My email's in my profile if you ever want to talk shop.