4 ms·
> It looks more enjoyable to use than BeautifulSoup. I don't believe BS is a full scraping solution, it's only the HTML parsing/querying isn't it? In that case
by danpalmer 9y ago
> It looks more enjoyable to use than BeautifulSoup.
I don't believe BS is a full scraping solution, it's only the HTML parsing/querying isn't it? In that case, this project actually uses lxml for that part - a relatively well known alternative to BS.
I highly recommend lxml, the API isn't perfect, but in my experience it's much more powerful than BS, and significantly faster as well. We run custom scrapers for a large number of websites, and apart from a few where we use JSON feeds, the majority use lxml, it has been very useful.
- tedmiston 9y agoYes, BeautifulSoup is focused on parsing not crawling (BS supports the lxml parser out of the box). Scrapy is more of an opinionated scraping framework whereas BS is a parsing library for scraping. I think the choice depends on what exactly you're trying to build and scale. I like both personally, though I'd use BS for simple MVPs and Scrapy if I wanted to crawl thousands of pages.