4 ms·

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server witho

by est 11y ago

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

foota 11y ago
I know there's extensions that help already with generating finders for automated e2e testing, you might be able to use something like that.
caseyf7 11y ago
The Resurrectio Chrome extension allows you to do this with PhantomJS
popey456963 11y ago
SeleniumIDE[0] provides a nice and simple way of doing this, it's just a very simple Firefox addon that lets you record and playback mouse movements, typing, etc. You can then improve your macro through Selenium WebDriver. [0] http://www.seleniumhq.org/ http://www.seleniumhq.org/
- enibundo 11y ago
  One problem with selenium last time I used it, was that it is very slow. Maybe this python library fixes this (i.e. no browser will show).
  - popey456963 11y ago
    Alas, this is both a downside and an upside of Selenium. It's rather slow because it does need to spin up a Firefox instance, but it is very user-friendly and easy to learn because you can see exactly where you are at just by looking at the web browser. You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others. The only reason this is (in my opinion), better than other solutions is because you can see the physical webpage it's loading, and for the sheer ease of use that this has. You don't even really need any coding experience to get a simple test running.
    - seanp2k2 11y ago
      You can record tests in Selenium or something like Capybara and replay them using something like PhantomJS, which is a headless browser-like JS execution environment that does things like generate a would-be DOM: https://github.com/jnicklas/capybara https://github.com/jnicklas/capybara You can also use Selenium tests with things like https://www.browserstack.com/automate https://www.browserstack.com/automate , where TL;DR they run your selenium test on dozens of browser + platform combinations and send you the results, like screenshots and any javascript errors. If you're familiar with CI stuff, you can see how powerful this has the potential to be. It's non-trivial but very possible to run your own cluster of selenium nodes as well; check out the official Selenium Grid: http://www.seleniumhq.org/projects/grid/ http://www.seleniumhq.org/projects/grid/
      - enibundo 11y ago
        what is CI? ...CLI?
        shrikant 11y ago
        Continuous Integration.
    - logn 11y ago
      You can use my project https://github.com/machinepublishers/jbrowserdriver https://github.com/machinepublishers/jbrowserdriver which can be both headless or (what's the opposite of headless?) have a full GUI. It runs on Java using Java's built-in browser and will take a second to warm up when an instance is created, but after doing so, you can reuse the existing browser since I added a reset() API. Performance is comparable to desktop browsers (slightly slower), and for every action it blocks until AJAX page loads finish.
      - archagon 11y ago
        Headful, I would think.
  - smellf 11y ago
    The only reason to use Selenium at all these days is because it can run your tests in multiple real browsers. If you don't need to test cross-browser support, then you can use phantomjs or one of the bazillions of other tools mentioned in this thread.
tsergiu 11y ago
Shameless plug, check out https://parsehub.com https://parsehub.com
- guilamu 11y ago
  Parsehub is pretty great, the free tier is interesting and the support is excellent.
cookiecaper 11y ago
Several startups have tried this. http://kimonolabs.com http://kimonolabs.com was is one I had experimented with and it was recently bought and shut down by Palantir.
- soared 11y ago
  I use this all the time, its excellent for less technical users. You can download a desktop version and still use it, despite it being shut down.
itsyogesh 11y ago
I would urge you to try nightmare[1], its a web scraper that uses electron as a headless browser, and they have a plugin called daydream[2] that records your webpage actions and convert it into a nightmare script. You have to do a couple of retries, but it does work. [1] https://github.com/segmentio/nightmare https://github.com/segmentio/nightmare [2] https://github.com/segmentio/daydream https://github.com/segmentio/daydream

by est 11y ago

foota 11y ago

I know there's extensions that help already with generating finders for automated e2e testing, you might be able to use something like that.

caseyf7 11y ago

The Resurrectio Chrome extension allows you to do this with PhantomJS

popey456963 11y ago

SeleniumIDE[0] provides a nice and simple way of doing this, it's just a very simple Firefox addon that lets you record and playback mouse movements, typing, etc. You can then improve your macro through Selenium WebDriver. [0] http://www.seleniumhq.org/ http://www.seleniumhq.org/

enibundo 11y ago
One problem with selenium last time I used it, was that it is very slow. Maybe this python library fixes this (i.e. no browser will show).
- popey456963 11y ago
  Alas, this is both a downside and an upside of Selenium. It's rather slow because it does need to spin up a Firefox instance, but it is very user-friendly and easy to learn because you can see exactly where you are at just by looking at the web browser. You can run headless Selenium, and speed it up by using a static Firefox instance, but even then it'll be maybe 2-3x slower than some of the others. The only reason this is (in my opinion), better than other solutions is because you can see the physical webpage it's loading, and for the sheer ease of use that this has. You don't even really need any coding experience to get a simple test running.
  - seanp2k2 11y ago
    You can record tests in Selenium or something like Capybara and replay them using something like PhantomJS, which is a headless browser-like JS execution environment that does things like generate a would-be DOM: https://github.com/jnicklas/capybara https://github.com/jnicklas/capybara You can also use Selenium tests with things like https://www.browserstack.com/automate https://www.browserstack.com/automate , where TL;DR they run your selenium test on dozens of browser + platform combinations and send you the results, like screenshots and any javascript errors. If you're familiar with CI stuff, you can see how powerful this has the potential to be. It's non-trivial but very possible to run your own cluster of selenium nodes as well; check out the official Selenium Grid: http://www.seleniumhq.org/projects/grid/ http://www.seleniumhq.org/projects/grid/
    - enibundo 11y ago
      what is CI? ...CLI?
      - shrikant 11y ago
        Continuous Integration.
  - logn 11y ago
    You can use my project https://github.com/machinepublishers/jbrowserdriver https://github.com/machinepublishers/jbrowserdriver which can be both headless or (what's the opposite of headless?) have a full GUI. It runs on Java using Java's built-in browser and will take a second to warm up when an instance is created, but after doing so, you can reuse the existing browser since I added a reset() API. Performance is comparable to desktop browsers (slightly slower), and for every action it blocks until AJAX page loads finish.
    - archagon 11y ago
      Headful, I would think.
- smellf 11y ago
  The only reason to use Selenium at all these days is because it can run your tests in multiple real browsers. If you don't need to test cross-browser support, then you can use phantomjs or one of the bazillions of other tools mentioned in this thread.

tsergiu 11y ago

Shameless plug, check out https://parsehub.com https://parsehub.com

guilamu 11y ago
Parsehub is pretty great, the free tier is interesting and the support is excellent.

cookiecaper 11y ago

Several startups have tried this. http://kimonolabs.com http://kimonolabs.com was is one I had experimented with and it was recently bought and shut down by Palantir.

soared 11y ago
I use this all the time, its excellent for less technical users. You can download a desktop version and still use it, despite it being shut down.

itsyogesh 11y ago

I would urge you to try nightmare[1], its a web scraper that uses electron as a headless browser, and they have a plugin called daydream[2] that records your webpage actions and convert it into a nightmare script. You have to do a couple of retries, but it does work. [1] https://github.com/segmentio/nightmare https://github.com/segmentio/nightmare [2] https://github.com/segmentio/daydream https://github.com/segmentio/daydream