12 ms·

Web Scraping with Electron

57 points by tazeg95 8y ago

SSchick 8y ago
Are there any other advantages over things like webdriver or puppeteer?
- sanxiyn 8y ago
  No, there aren't. Just use WebDriver.
- nkozyra 8y ago
  Not really. I also have no idea what cheerio brings to the table here. Seems like a hefty solution to web scraping
- galacticdessert 8y ago
  One thing about Selenium and Puppeteer is that they trigger captcahs on some websites, making scraping impossible. This would perhaps fix it?
TicklishTiger 8y ago
I wish there was an easy way to send commands to the console of a browser. That would be all I need to satisfy all my browser automation tasks. Without installing and learning any frameworks. Say there was a linux command 'SendToChromium' that would do that for Chromium. Then to navigate to some page one could simply do this: SendToChromium location.href="/somepage.html" SendToChromium should return the output of the command. So to get the html of the current page, one would simply do: SendToChromium document.body.innerHTML > theBody.html Ideally the browser would listen for this type of command on a local port. So instead of needing a binary 'SendToChromium' one could simply start Chromium in listening mode: chromium --listen 12345 And then talk to it via http: curl 127.0.0.1:12345/execute?command=location.href="/somepage.html"
- dstick 8y ago
  What would you do with this solution that can’t already be done with headless browsers, apart from looking at it? It’s very much like that already: write script, send to browser, let it do it’s thing, run javascript code if you want and get the final renderend HTML and console output.
  - TicklishTiger 8y ago
    What would you do with this solution Automate the browser that can’t already be done I did not say it can't be done now. But I don't want the overhead of pupeteer, selenium plus the client libraries to control them. These things are fricking monsters. And they will go out of fashion at some point. Simple javascript commands will not. with headless browsers I don't want headless. I just want to automate.
    - phiresky 8y ago
      You can run puppeteer with any chromium binary, including non-headless mode. You can also just start chromium with debugging active `--remote-debugging-port=9222` and then connect to that port and send any dev console commands.
      - TicklishTiger 8y ago
        How do you send dev console commands to that port?
        phiresky 8y ago
        It uses the devtools protocol, see here https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Looks like it's a REST protocol. You have to connect to a tab and then call e.g. Runtime.evaluate {expression: "console.log('test')"}
        phiresky 8y ago
        Also, you can apparently connect the puppeteer client to a running chrome instance [1]. That way you can use the nice and statically typed puppeteer api and you should have pretty much zero overhead. [1] https://github.com/GoogleChrome/puppeteer/issues/238 https://github.com/GoogleChrome/puppeteer/issues/238
Dunedan 8y ago
> Is there a better way to surf the web, retrieve the source code of the pages and extract data from them ? Yes, of course! To get the source code of a web site you don't need a browser and all its complexity. It makes me so sad how far we have come in terms of unnecessary complexity for simple tasks. If you want to extract data from web pages without requiring hundreds of megabytes for something like Electron, there are lots of scraping libraries out there. There are for example at least two good Python implementations: Scrapy[1] and BeautifulSoup[2]. [1]: https://scrapy.org/ https://scrapy.org/ [2]: https://www.crummy.com/software/BeautifulSoup/ https://www.crummy.com/software/BeautifulSoup/
- dstick 8y ago
  Correct me if I’m wrong but neither one supports Javascript rendered pages? You’re right in the overhead though; I’d stay miles away from Electron for scraping but you’ll need more than a CURL wrapper to properly fetch data in all shapes and sizes :) Headless Chromium does do the trick in that regard.
  - danpalmer 8y ago
    With web scraping you typically don’t want the visuals anyway. JS rendered applications are usually easier to scrape because they have data in a more raw or canonical format available somewhere to do that rendering.
    - karmelapple 8y ago
      Plenty of websites will only render the content fully after some JavaScript runs, so to properly scrape them you do indeed need a browser to process the JS. This includes text content.
      - adamzochowski 8y ago
        Javascript rendered pages load JS which then in turn calls some rest API to get data and use that to render contents. Web scraper stops scraping the html, but calls and scrapes the rest api endpoint.
aboutruby 8y ago
Interesting but seems less powerful than my current setup: - I have mitmproxy to capture the traffic / manipulate the traffic - I have Chrome opened with Selenium/Capybara/chromedriver and using mitmproxy - I then browse to the target pages, it records the selected requests and the selected responses - It then replays the requests until they fail (with a delay) I highly recommend mitmproxy, it's extremely powerful: capture traffic, send responses without hitting the server, block/hang requests, modify responses, modify requests/responses headers. Then higher level interfaces can be built on top, Selenium allows you to load Chrome extensions and execute Javascript on any page for instance. You can also manage many tabs at the same time. I could make a blog post/demo if people are interested
- marvel_boy 8y ago
  >I could make a blog post/demo if people are interested yes, please !
- kuhhk 8y ago
  A blog article would be nice. Sounds interesting, but I’m having a hard-time understanding. If it’s replaying requests, how do you get it to do things like go to the next pagination and click on all of the next paginated results?
  - aboutruby 8y ago
    In my case I can't do the pagination automatically so I have to fetch the pages myself to then have them replayed. In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API). You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows. The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect). Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that. Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev https://twitter.com/localhostdotdev (that I will turn into a blog).
CGamesPlay 8y ago
I'm going to plug my app that does scraping with Electron: https://github.com/CGamesPlay/chronicler https://github.com/CGamesPlay/chronicler To the commenters who don't understand why this is necessary: - It reliably loads linked resources in a WYSIWYG fashion, including embedded media and other things that have to be handled in an ad-hoc fashion when using something like BeautifulSoup. - It handles resources loaded through JavaScript, including HTML5 History API changes.
- aloer 8y ago
  Could you explain what electron offers here over for example a browser plugin? I'm not that familiar with limitations of WebExtension APIs It looks like an interesting project but only for a few selected sites. For more random browsing I believe I would be too security conscious (https://electronjs.org/docs/tutorial/security https://electronjs.org/docs/tutorial/security) to allow it Might be unlikely that some random script on a random site will target Electron but you never know
  - CGamesPlay 8y ago
    Well in Chrome you can't hook into the network layer to record/replay requests like I've done here (you could fake it by overriding the ServiceWorker, possibly, but this feels brittle and I'm not sure if it's possible either). I'm not familiar with Firefox extensions but I'm given to understand you likely could implement this project as a Firefox extension. Locking down Electron apps to be safe on the larger web is certainly one area where I think electron could do a lot better. I think my project has followed all of the recommendations and should be safe, but I agree with you that it feels like a bigger attack surface. I personally wanted it to support offline browsing of documentation sites, which are generally pretty "safe" from that perspective.

57 points by tazeg95 8y ago

SSchick 8y ago

Are there any other advantages over things like webdriver or puppeteer?

sanxiyn 8y ago
No, there aren't. Just use WebDriver.
nkozyra 8y ago
Not really. I also have no idea what cheerio brings to the table here. Seems like a hefty solution to web scraping
galacticdessert 8y ago
One thing about Selenium and Puppeteer is that they trigger captcahs on some websites, making scraping impossible. This would perhaps fix it?

TicklishTiger 8y ago

I wish there was an easy way to send commands to the console of a browser. That would be all I need to satisfy all my browser automation tasks. Without installing and learning any frameworks. Say there was a linux command 'SendToChromium' that would do that for Chromium. Then to navigate to some page one could simply do this: SendToChromium location.href="/somepage.html" SendToChromium should return the output of the command. So to get the html of the current page, one would simply do: SendToChromium document.body.innerHTML > theBody.html Ideally the browser would listen for this type of command on a local port. So instead of needing a binary 'SendToChromium' one could simply start Chromium in listening mode: chromium --listen 12345 And then talk to it via http: curl 127.0.0.1:12345/execute?command=location.href="/somepage.html"

dstick 8y ago
What would you do with this solution that can’t already be done with headless browsers, apart from looking at it? It’s very much like that already: write script, send to browser, let it do it’s thing, run javascript code if you want and get the final renderend HTML and console output.
- TicklishTiger 8y ago
  What would you do with this solution Automate the browser that can’t already be done I did not say it can't be done now. But I don't want the overhead of pupeteer, selenium plus the client libraries to control them. These things are fricking monsters. And they will go out of fashion at some point. Simple javascript commands will not. with headless browsers I don't want headless. I just want to automate.
  - phiresky 8y ago
    You can run puppeteer with any chromium binary, including non-headless mode. You can also just start chromium with debugging active `--remote-debugging-port=9222` and then connect to that port and send any dev console commands.
    - TicklishTiger 8y ago
      How do you send dev console commands to that port?
      - phiresky 8y ago
        It uses the devtools protocol, see here https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Looks like it's a REST protocol. You have to connect to a tab and then call e.g. Runtime.evaluate {expression: "console.log('test')"}
      - phiresky 8y ago
        Also, you can apparently connect the puppeteer client to a running chrome instance [1]. That way you can use the nice and statically typed puppeteer api and you should have pretty much zero overhead. [1] https://github.com/GoogleChrome/puppeteer/issues/238 https://github.com/GoogleChrome/puppeteer/issues/238

Dunedan 8y ago

> Is there a better way to surf the web, retrieve the source code of the pages and extract data from them ? Yes, of course! To get the source code of a web site you don't need a browser and all its complexity. It makes me so sad how far we have come in terms of unnecessary complexity for simple tasks. If you want to extract data from web pages without requiring hundreds of megabytes for something like Electron, there are lots of scraping libraries out there. There are for example at least two good Python implementations: Scrapy[1] and BeautifulSoup[2]. [1]: https://scrapy.org/ https://scrapy.org/ [2]: https://www.crummy.com/software/BeautifulSoup/ https://www.crummy.com/software/BeautifulSoup/

dstick 8y ago
Correct me if I’m wrong but neither one supports Javascript rendered pages? You’re right in the overhead though; I’d stay miles away from Electron for scraping but you’ll need more than a CURL wrapper to properly fetch data in all shapes and sizes :) Headless Chromium does do the trick in that regard.
- danpalmer 8y ago
  With web scraping you typically don’t want the visuals anyway. JS rendered applications are usually easier to scrape because they have data in a more raw or canonical format available somewhere to do that rendering.
  - karmelapple 8y ago
    Plenty of websites will only render the content fully after some JavaScript runs, so to properly scrape them you do indeed need a browser to process the JS. This includes text content.
    - adamzochowski 8y ago
      Javascript rendered pages load JS which then in turn calls some rest API to get data and use that to render contents. Web scraper stops scraping the html, but calls and scrapes the rest api endpoint.

aboutruby 8y ago

Interesting but seems less powerful than my current setup: - I have mitmproxy to capture the traffic / manipulate the traffic - I have Chrome opened with Selenium/Capybara/chromedriver and using mitmproxy - I then browse to the target pages, it records the selected requests and the selected responses - It then replays the requests until they fail (with a delay) I highly recommend mitmproxy, it's extremely powerful: capture traffic, send responses without hitting the server, block/hang requests, modify responses, modify requests/responses headers. Then higher level interfaces can be built on top, Selenium allows you to load Chrome extensions and execute Javascript on any page for instance. You can also manage many tabs at the same time. I could make a blog post/demo if people are interested

marvel_boy 8y ago
>I could make a blog post/demo if people are interested yes, please !
kuhhk 8y ago
A blog article would be nice. Sounds interesting, but I’m having a hard-time understanding. If it’s replaying requests, how do you get it to do things like go to the next pagination and click on all of the next paginated results?
- aboutruby 8y ago
  In my case I can't do the pagination automatically so I have to fetch the pages myself to then have them replayed. In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API). You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows. The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect). Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that. Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev https://twitter.com/localhostdotdev (that I will turn into a blog).

CGamesPlay 8y ago

I'm going to plug my app that does scraping with Electron: https://github.com/CGamesPlay/chronicler https://github.com/CGamesPlay/chronicler To the commenters who don't understand why this is necessary: - It reliably loads linked resources in a WYSIWYG fashion, including embedded media and other things that have to be handled in an ad-hoc fashion when using something like BeautifulSoup. - It handles resources loaded through JavaScript, including HTML5 History API changes.

aloer 8y ago
Could you explain what electron offers here over for example a browser plugin? I'm not that familiar with limitations of WebExtension APIs It looks like an interesting project but only for a few selected sites. For more random browsing I believe I would be too security conscious (https://electronjs.org/docs/tutorial/security https://electronjs.org/docs/tutorial/security) to allow it Might be unlikely that some random script on a random site will target Electron but you never know
- CGamesPlay 8y ago
  Well in Chrome you can't hook into the network layer to record/replay requests like I've done here (you could fake it by overriding the ServiceWorker, possibly, but this feels brittle and I'm not sure if it's possible either). I'm not familiar with Firefox extensions but I'm given to understand you likely could implement this project as a Firefox extension. Locking down Electron apps to be safe on the larger web is certainly one area where I think electron could do a lot better. I think my project has followed all of the recommendations and should be safe, but I agree with you that it feels like a bigger attack surface. I personally wanted it to support offline browsing of documentation sites, which are generally pretty "safe" from that perspective.