4 ms·
I used BeautifulSoup for a project once because of all the accolades it gets here but found it to be less than robust. It might be sufficient in a scenario whe
by pmorici 14y ago
I used BeautifulSoup for a project once because of all the accolades it gets here but found it to be less than robust. It might be sufficient in a scenario where you have a single site you need to scrap but I found it was totally unreliable when used across a wide range of sites and esp. sites with foreign language content.
- ludwigvan 14y agoI started using PyQuery yesterday, after using BeautifulSoup for a long time. It seems much easier to use. pq_page = pquery(url=PAGE_URL) Note that PyQuery has some encoding issues too (or rather the sites I were scraping were too bad, showing two different encodings in meta tag!), here are two different things I have done to workaround: page = requests.get(PAGE_URL) pq_page = pquery(page.text) If that doesn't cut it (because requests detects it wrong too), try forcing the encoding in requests: page = requests.get(PAGE_URL) page.encoding = 'utf-8' pq_page = pquery(page.text)