Absolutely love BeautifulSoup, I use it almost everyday in my job. There isn't really a scrapy vs. BS4 divide as you can still use the library with Scrapy, as o

Absolutely love BeautifulSoup, I use it almost everyday in my job. There isn't really a scrapy vs. BS4 divide as you can still use the library with Scrapy, as opposed to it's standard parsing functionality. It also works well with lxml which is considerably faster. It's also possible to build very performant and large scale crawlers with just BS4 and requests . Though managing the architecture is a bit of a pain, but it definitely can be done. There are also a number of cases where it's better to use Bs4 than scrapy. Also using PhantomJS, Selenium and BS4 can provide you with a very powerful data scraping solution.