3 ms·
One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on l
by xfour 13y ago
One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on lxml) so much easier with parsing the scraped data.
- brechin 13y agoI'd consider it still active, since it was updated on 2013-06-07: https://pypi.python.org/pypi/beautifulsoup4 https://pypi.python.org/pypi/beautifulsoup4 I prefer using lxml myself, since I like using XPath queries, but bs4 sometimes parses broken HTML better than any of the provided lxml parsers do.
- ianhawes 13y agoSomething to consider is that the trend the past year has been to use headless browsers over BeautifulSoup, cURL, etc.. because headless browsers are harder to detect by anti-scraping systems and can interpret JavaScript.
- takluyver 13y agoThat's what the OP is about ;-). But BeautifulSoup isn't a way to retrieve a web page, it's a way to parse HTML. You can get the page with a headless browser, and then transfer the DOM into a BeautifulSoup tree to do your scraping.
- takluyver 13y agoBS4, which is still actively developed, got out of the parser game - it can now use lxml (fast) or html5lib (highly tolerant) to parse the HTML. It's kept the convenient interface to dig into the DOM, and it's kept the UnicodeDammit encoding detection system.