4 ms·

> Is there a better way to surf the web, retrieve the source code of the pages and extract data from them ? Yes, of course! To get the source code of a web sit

> Is there a better way to surf the web, retrieve the source code of the pages and extract data from them ? Yes, of course! To get the source code of a web site you don't need a browser and all its complexity. It makes me so sad how far we have come in terms of unnecessary complexity for simple tasks. If you want to extract data from web pages without requiring hundreds of megabytes for something like Electron, there are lots of scraping libraries out there. There are for example at least two good Python implementations: Scrapy[1] and BeautifulSoup[2]. [1]: https://scrapy.org/ https://scrapy.org/ [2]: https://www.crummy.com/software/BeautifulSoup/ https://www.crummy.com/software/BeautifulSoup/

dstick 8y ago
Correct me if I’m wrong but neither one supports Javascript rendered pages? You’re right in the overhead though; I’d stay miles away from Electron for scraping but you’ll need more than a CURL wrapper to properly fetch data in all shapes and sizes :) Headless Chromium does do the trick in that regard.
- danpalmer 8y ago
  With web scraping you typically don’t want the visuals anyway. JS rendered applications are usually easier to scrape because they have data in a more raw or canonical format available somewhere to do that rendering.
  - karmelapple 8y ago
    Plenty of websites will only render the content fully after some JavaScript runs, so to properly scrape them you do indeed need a browser to process the JS. This includes text content.
    - adamzochowski 8y ago
      Javascript rendered pages load JS which then in turn calls some rest API to get data and use that to render contents. Web scraper stops scraping the html, but calls and scrapes the rest api endpoint.
tazeg95 8y ago
Sure, but i meant to build a portable app, for end users who are not coders, with a GUI, and for a dedicated purpose, like for exemple navigating on facebook. So i will edit this question to this : Is there a better way to code a portable application with a graphical user interface to scrape a given site ? Thanks for your comment.
- chinathrow 8y ago
  > like for exemple navigating on facebook. What would you want to scrape there which is not against their ToS and a violation of user privacy in general?
  - tyingq 8y ago
    I wouldn't mind a personal scraper that pulls down the family updates and pictures I want and puts them somewhere private where I can see them. Would get rid of the clutter and keep FB from some amount of shenanigans with my browser.
  - Nextgrid 8y ago
    I can’t believe there are people still defending this scummy company. Facebook broke both legal and ethical “ToS” countless times and has no plan to stop. Why do you consider what Facebook is doing as OK but a little web scraping for personal usage to be so bad?
    - leetbulb 8y ago
      As the saying goes: "two wrongs don't make a right." Facebook's ToS is still a ToS. If you want to scrape the data that they've collected, either risk your account due to it being against the ToS or collect the data yourself.
- rasengan 8y ago
  You can access the html of the website and use regular expressions.
bootloop 8y ago
I mean, I guess the point is that it allows you to scrap data after it was rendered by JS.
- patrickm1 8y ago
  you can always use something like proxycrawl to scrape javascript without using Electron. And it's compatible with scrapy
IloveHN84 8y ago
You forgot curl in C/C++ which is the most advanced tool out there
fake-name 8y ago
This is nice sounding, but many modern web-pages use extensive client-side rendering. Sure, you can work around that without needing a full JS environment, but doing so is ad-hoc and you wind up having to write complex code on a per-site basis. I do a bunch of web-scraping for hobby shit, and I've love to be able to not have to shell out to chromium for some sites, but unfortunately the modern web basically means you're stuck with it.
- harryf 8y ago
  Also sites with some kind of 2FA / oauth happening. This _looks_ like it would be possible to login manually then start scraping.
dig1 8y ago
Good luck with that :) Any modern website requires javascript interpreter on client side, so unless you provide some sort of javascript interpretation (which can be messy), you'll be able to scrape only simple content with scrapy/BS.

dstick 8y ago

Correct me if I’m wrong but neither one supports Javascript rendered pages? You’re right in the overhead though; I’d stay miles away from Electron for scraping but you’ll need more than a CURL wrapper to properly fetch data in all shapes and sizes :) Headless Chromium does do the trick in that regard.

danpalmer 8y ago
With web scraping you typically don’t want the visuals anyway. JS rendered applications are usually easier to scrape because they have data in a more raw or canonical format available somewhere to do that rendering.
- karmelapple 8y ago
  Plenty of websites will only render the content fully after some JavaScript runs, so to properly scrape them you do indeed need a browser to process the JS. This includes text content.
  - adamzochowski 8y ago
    Javascript rendered pages load JS which then in turn calls some rest API to get data and use that to render contents. Web scraper stops scraping the html, but calls and scrapes the rest api endpoint.

tazeg95 8y ago

Sure, but i meant to build a portable app, for end users who are not coders, with a GUI, and for a dedicated purpose, like for exemple navigating on facebook. So i will edit this question to this : Is there a better way to code a portable application with a graphical user interface to scrape a given site ? Thanks for your comment.

chinathrow 8y ago
> like for exemple navigating on facebook. What would you want to scrape there which is not against their ToS and a violation of user privacy in general?
- tyingq 8y ago
  I wouldn't mind a personal scraper that pulls down the family updates and pictures I want and puts them somewhere private where I can see them. Would get rid of the clutter and keep FB from some amount of shenanigans with my browser.
- Nextgrid 8y ago
  I can’t believe there are people still defending this scummy company. Facebook broke both legal and ethical “ToS” countless times and has no plan to stop. Why do you consider what Facebook is doing as OK but a little web scraping for personal usage to be so bad?
  - leetbulb 8y ago
    As the saying goes: "two wrongs don't make a right." Facebook's ToS is still a ToS. If you want to scrape the data that they've collected, either risk your account due to it being against the ToS or collect the data yourself.
rasengan 8y ago
You can access the html of the website and use regular expressions.

bootloop 8y ago

I mean, I guess the point is that it allows you to scrap data after it was rendered by JS.

patrickm1 8y ago
you can always use something like proxycrawl to scrape javascript without using Electron. And it's compatible with scrapy

IloveHN84 8y ago

You forgot curl in C/C++ which is the most advanced tool out there

fake-name 8y ago

This is nice sounding, but many modern web-pages use extensive client-side rendering. Sure, you can work around that without needing a full JS environment, but doing so is ad-hoc and you wind up having to write complex code on a per-site basis. I do a bunch of web-scraping for hobby shit, and I've love to be able to not have to shell out to chromium for some sites, but unfortunately the modern web basically means you're stuck with it.

harryf 8y ago
Also sites with some kind of 2FA / oauth happening. This _looks_ like it would be possible to login manually then start scraping.

dig1 8y ago

Good luck with that :) Any modern website requires javascript interpreter on client side, so unless you provide some sort of javascript interpretation (which can be messy), you'll be able to scrape only simple content with scrapy/BS.