4 ms·

How to scrape and extract hyperlink networks with BeautifulSoup and NetworkX

53 points by spacejunkjim 5y ago

rabuse 5y ago
This breaks with using standard web scraping methods (non-headless JS engines). Had to deal with this issue recently due to everything being a damn SPA now. Look into Selenium for running headless browsers if looking to scrape the modern web.
- ricardo81 5y ago
  I used to use mozrepl back in the day before FF Quantum (or a version nearby) broke it, worked great- run the browser(s) via telnet. Could run as many browsers as memory would allow.
- kordlessagain 5y ago
  Depending on the use case you might try imaging the page, then send the image to an ML model for full text before indexing. If you need links extracted, Selenium also supports parsing the assembled DOM: https://github.com/kordless/grub-2.0/tree/main/aperture https://github.com/kordless/grub-2.0/tree/main/aperture
- bshipp 5y ago
  While I certainly don't disagree that many sites need selenium these days, I find it's often possible to go straight to the AJAX/xHR source and get a pre-parsed JSON with all the data you're looking for.
funnyflamigo 5y ago
I know some people think all scraping is bad or malicious. I'd like to point out this is a perfectly legitimate use case for it, in fact this is how Google Search operates. Web scraping done correctly should be barely noticeable if at all to the operators. Don't send 10,000 req/s, have aggressive delays, make your retries extremely generous, try to avoid pages or actions you know are "heavy". You don't need to update data from every product page every 5 minutes.
- tyingq 5y ago
  My guess is that scraping is getting heavier because scrapers have to use headless browsers now. And so, probably downloading artifacts they don't need...because they can't tell what's needed or not, at least with js.
pjfin123 5y ago
I wrote a similar Python library to do Beautiful Soup scraping with basic PageRank and a Flask web app: https://github.com/argosopentech/argos-search https://github.com/argosopentech/argos-search

53 points by spacejunkjim 5y ago

This breaks with using standard web scraping methods (non-headless JS engines). Had to deal with this issue recently due to everything being a damn SPA now. Look into Selenium for running headless browsers if looking to scrape the modern web.

ricardo81 5y ago
I used to use mozrepl back in the day before FF Quantum (or a version nearby) broke it, worked great- run the browser(s) via telnet. Could run as many browsers as memory would allow.
kordlessagain 5y ago
Depending on the use case you might try imaging the page, then send the image to an ML model for full text before indexing. If you need links extracted, Selenium also supports parsing the assembled DOM: https://github.com/kordless/grub-2.0/tree/main/aperture https://github.com/kordless/grub-2.0/tree/main/aperture
bshipp 5y ago
While I certainly don't disagree that many sites need selenium these days, I find it's often possible to go straight to the AJAX/xHR source and get a pre-parsed JSON with all the data you're looking for.

funnyflamigo 5y ago

I know some people think all scraping is bad or malicious. I'd like to point out this is a perfectly legitimate use case for it, in fact this is how Google Search operates. Web scraping done correctly should be barely noticeable if at all to the operators. Don't send 10,000 req/s, have aggressive delays, make your retries extremely generous, try to avoid pages or actions you know are "heavy". You don't need to update data from every product page every 5 minutes.

tyingq 5y ago
My guess is that scraping is getting heavier because scrapers have to use headless browsers now. And so, probably downloading artifacts they don't need...because they can't tell what's needed or not, at least with js.

pjfin123 5y ago

I wrote a similar Python library to do Beautiful Soup scraping with basic PageRank and a Flask web app: https://github.com/argosopentech/argos-search https://github.com/argosopentech/argos-search