5 ms·

For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster. Also, I think very few people use MechanicalSoup nowadays. There are

by f311a 5y ago

For Python, instead of BeautifulSoup I prefer to use selectolax which is 3-5 times faster. Also, I think very few people use MechanicalSoup nowadays. There are libraries that allow you to use headless Chrome, e.g. Playwright. It looks like the author of the article just googled some libraries for each language and didn't research the topic.

mdaniel 5y ago
Lazyweb link: https://github.com/rushter/selectolax https://github.com/rushter/selectolax although I don't follow the need to have what appears to be two completely separate HTML parsing C libraries as dependencies; seeing this in the readme for Modest gives me the shivers because lxml has _seen some shit_ > Modest is a fast HTML renderer implemented as a pure C99 library with no outside dependencies. although its other dep seems much more cognizant about the HTML5 standard, for whatever that's worth: https://github.com/lexbor/lexbor#lexbor https://github.com/lexbor/lexbor#lexbor --- > It looks like the author of the article just googled some libraries for each language and didn't research the topic Heh, oh, new to the Internet, are you? :-D
jacurtis 5y ago
> It looks like the author of the article just googled some libraries for each language and didn't research the topic Yep, this seemed like an aggregate Google results page. I was initially intrigued by the article and then realized it was a list of libraries the author found via Google. There were significantly notable omissions from this list and a bunch of weird stuff that feels unnecessary. I don't think the author has actually scraped a page before.
xnyan 5y ago
I agree with your conclusion, but in any discussion about web scraping it's probably a good idea to mention BeautifulSoup given how popular it is (virtually a builtin in terms how much it's used) and given all the documentation available for it, a good starting point if perf is not going to be a concern.
heavyset_go 5y ago
requests-html is faster than bs4 using lxml. It's a wrapper over lxml. I built something similar years ago using a similar method, it was much faster than bs4, too.

by f311a 5y ago

mdaniel 5y ago

Lazyweb link: https://github.com/rushter/selectolax https://github.com/rushter/selectolax although I don't follow the need to have what appears to be two completely separate HTML parsing C libraries as dependencies; seeing this in the readme for Modest gives me the shivers because lxml has _seen some shit_ > Modest is a fast HTML renderer implemented as a pure C99 library with no outside dependencies. although its other dep seems much more cognizant about the HTML5 standard, for whatever that's worth: https://github.com/lexbor/lexbor#lexbor https://github.com/lexbor/lexbor#lexbor --- > It looks like the author of the article just googled some libraries for each language and didn't research the topic Heh, oh, new to the Internet, are you? :-D

jacurtis 5y ago

> It looks like the author of the article just googled some libraries for each language and didn't research the topic Yep, this seemed like an aggregate Google results page. I was initially intrigued by the article and then realized it was a list of libraries the author found via Google. There were significantly notable omissions from this list and a bunch of weird stuff that feels unnecessary. I don't think the author has actually scraped a page before.

xnyan 5y ago

I agree with your conclusion, but in any discussion about web scraping it's probably a good idea to mention BeautifulSoup given how popular it is (virtually a builtin in terms how much it's used) and given all the documentation available for it, a good starting point if perf is not going to be a concern.

heavyset_go 5y ago

requests-html is faster than bs4 using lxml. It's a wrapper over lxml. I built something similar years ago using a similar method, it was much faster than bs4, too.