5 ms·
This looks quite promising and I look forward to trying it out. I wonder how it will work compared with my current approach of using Mechanize and BeautifulSoup
by lunchbox 18y ago
This looks quite promising and I look forward to trying it out. I wonder how it will work compared with my current approach of using Mechanize and BeautifulSoup, along with the threading module.
- tdavis 18y agoFor long-running, large jobs I can tell you from experience it would work about a gazillion times faster. Especially if you drop BeautifulSoup for something like lxml.
- breck 18y agoAgreed. BeautifulSoup can be quite slow(unless I'm doing it wrong, which I probably am). It accounts for 98% of the time of a current job I'm running. If anyone can provide some tips it'd be much appreciated.
- tdavis 18y agoIt's rather unlikely you're doing anything wrong. IIRC, BeautifulSoup docs acknowledge that it is rather slow. Also, if you don't make use of the methods that extract/unravel object trees, they may not be properly GC'd, leading to further slowdowns. I can't remember the method names exactly (might be destroy() and extract()), but they're in the docs.
- iamelgringo 18y agoI can't speak about Mechanize, but I did a project with BeautifulSoup a few months back. For scraping specific elements in a page, the xpath/Firebug integration is a huge win. Being able to highlight an item and grab the xpath selector in Firebug saves so much time, it's not even funny.