3 ms·
For long-running, large jobs I can tell you from experience it would work about a gazillion times faster. Especially if you drop BeautifulSoup for something lik
by tdavis 18y ago
For long-running, large jobs I can tell you from experience it would work about a gazillion times faster. Especially if you drop BeautifulSoup for something like lxml.
- breck 18y agoAgreed. BeautifulSoup can be quite slow(unless I'm doing it wrong, which I probably am). It accounts for 98% of the time of a current job I'm running. If anyone can provide some tips it'd be much appreciated.
- tdavis 18y agoIt's rather unlikely you're doing anything wrong. IIRC, BeautifulSoup docs acknowledge that it is rather slow. Also, if you don't make use of the methods that extract/unravel object trees, they may not be properly GC'd, leading to further slowdowns. I can't remember the method names exactly (might be destroy() and extract()), but they're in the docs.