5 ms·

I wish there was an easy way to send commands to the console of a browser. That would be all I need to satisfy all my browser automation tasks. Without instal

I wish there was an easy way to send commands to the console of a browser. That would be all I need to satisfy all my browser automation tasks. Without installing and learning any frameworks. Say there was a linux command 'SendToChromium' that would do that for Chromium. Then to navigate to some page one could simply do this: SendToChromium location.href="/somepage.html" SendToChromium should return the output of the command. So to get the html of the current page, one would simply do: SendToChromium document.body.innerHTML > theBody.html Ideally the browser would listen for this type of command on a local port. So instead of needing a binary 'SendToChromium' one could simply start Chromium in listening mode: chromium --listen 12345 And then talk to it via http: curl 127.0.0.1:12345/execute?command=location.href="/somepage.html"

dstick 8y ago
What would you do with this solution that can’t already be done with headless browsers, apart from looking at it? It’s very much like that already: write script, send to browser, let it do it’s thing, run javascript code if you want and get the final renderend HTML and console output.
- TicklishTiger 8y ago
  What would you do with this solution Automate the browser that can’t already be done I did not say it can't be done now. But I don't want the overhead of pupeteer, selenium plus the client libraries to control them. These things are fricking monsters. And they will go out of fashion at some point. Simple javascript commands will not. with headless browsers I don't want headless. I just want to automate.
  - phiresky 8y ago
    You can run puppeteer with any chromium binary, including non-headless mode. You can also just start chromium with debugging active `--remote-debugging-port=9222` and then connect to that port and send any dev console commands.
    - TicklishTiger 8y ago
      How do you send dev console commands to that port?
      - phiresky 8y ago
        It uses the devtools protocol, see here https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Looks like it's a REST protocol. You have to connect to a tab and then call e.g. Runtime.evaluate {expression: "console.log('test')"}
      - phiresky 8y ago
        Also, you can apparently connect the puppeteer client to a running chrome instance [1]. That way you can use the nice and statically typed puppeteer api and you should have pretty much zero overhead. [1] https://github.com/GoogleChrome/puppeteer/issues/238 https://github.com/GoogleChrome/puppeteer/issues/238
  - matthewmacleod 8y ago
    But I don't want the overhead of pupeteer I don't understand this objection. Puppeteer is the client library that you want, and it's not particularly massive. It's the plumbing to handle all of the stuff around starting and connecting to browsers, providing a tidy API to abstract the stuff that's annoying to use directly. In fact, you can do almost anything you want to using the existing interface that Chrome has built in: chromium --remote-debugging-port=9222 You can connect to it using e.g. the WS command line tool using the URL it spits out: ws ws://127.0.0.1:9222/devtools/browser/81b9b178-ece0-4953-8ea0-bce6ac31d89c Then you can make it do things: {"id": 1, "method": "Target.createTarget", "params": {"url": "http://www.google.com"}} It does what you want, over a web socket interface rather than HTTP.
    - TicklishTiger 8y ago
      That looks interesting. Where does that ws tool come from? Is it this one? https://packages.debian.org/stretch/node-ws https://packages.debian.org/stretch/node-ws How do you send javascript commands over that protocol?
  - deleted 8y ago
    [deleted]
androidgirl 8y ago
You can do this with Selenium pretty easily, and low level webdrivers support it too.
- TicklishTiger 8y ago
  I would give low level webdrivers a go. But so far I have not even figured out how to install one for Chromium on Debian. apt install chromedriver gives me: Package 'chromedriver' has no installation candidate There is something called "chromium-chromedriver". Let me try that ... one moment ... Ok. So I start it via: apt install chromium-chromedriver Now according to the docs, this should create a browser: curl -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }' http://localhost:9515/session Ha! It works! So chromedriver might be a solution!
SSchick 8y ago
What you are describing is exactly what puppeteer uses internally, you might want to explore their code base.
dlkinney 8y ago
While not currently "easy", there exists the Chrome Devtools Protocol.[0] I'm not aware of a CLI utility that communicates with it, but it wouldn't be impossible to make one that fulfills what you're looking for. A second tool could then act as a REST proxy, if calling the commands via curl is really your jam. I think you've given my weekend some purpose. Lemme see what I can pull together... [0] https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/
- TicklishTiger 8y ago
  Yes, that might work. Maybe an even better approach is to use chromium-chromedriver. I just got it working like this: apt install chromium-chromedriver chromedriver This seems to create a service that listens on port 9515 for standardized commands to remote control a chromium instance. The commands seem to be specified by the W3C: https://www.w3.org/TR/webdriver/ https://www.w3.org/TR/webdriver/ I got it to open a browser with this curl command: curl -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }' http://localhost:9515/session I have not yet figured out how to send javascript commands though.
aasasd 8y ago
With ‘native messaging,’ you can have your program communicate with your extension, so the extension could then do everything that's available to it via the API. Won't be surprised if an extension like this already exists.
fake-name 8y ago
I've written something[1] that can basically do this, though it's non-interactive. The CLI interface is somewhat incomplete as-of-now, but it'd be fairly easy to add more comprehensive tools. https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController

dstick 8y ago

What would you do with this solution that can’t already be done with headless browsers, apart from looking at it? It’s very much like that already: write script, send to browser, let it do it’s thing, run javascript code if you want and get the final renderend HTML and console output.

TicklishTiger 8y ago
What would you do with this solution Automate the browser that can’t already be done I did not say it can't be done now. But I don't want the overhead of pupeteer, selenium plus the client libraries to control them. These things are fricking monsters. And they will go out of fashion at some point. Simple javascript commands will not. with headless browsers I don't want headless. I just want to automate.
- phiresky 8y ago
  You can run puppeteer with any chromium binary, including non-headless mode. You can also just start chromium with debugging active `--remote-debugging-port=9222` and then connect to that port and send any dev console commands.
  - TicklishTiger 8y ago
    How do you send dev console commands to that port?
    - phiresky 8y ago
      It uses the devtools protocol, see here https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/ Looks like it's a REST protocol. You have to connect to a tab and then call e.g. Runtime.evaluate {expression: "console.log('test')"}
    - phiresky 8y ago
      Also, you can apparently connect the puppeteer client to a running chrome instance [1]. That way you can use the nice and statically typed puppeteer api and you should have pretty much zero overhead. [1] https://github.com/GoogleChrome/puppeteer/issues/238 https://github.com/GoogleChrome/puppeteer/issues/238
- matthewmacleod 8y ago
  But I don't want the overhead of pupeteer I don't understand this objection. Puppeteer is the client library that you want, and it's not particularly massive. It's the plumbing to handle all of the stuff around starting and connecting to browsers, providing a tidy API to abstract the stuff that's annoying to use directly. In fact, you can do almost anything you want to using the existing interface that Chrome has built in: chromium --remote-debugging-port=9222 You can connect to it using e.g. the WS command line tool using the URL it spits out: ws ws://127.0.0.1:9222/devtools/browser/81b9b178-ece0-4953-8ea0-bce6ac31d89c Then you can make it do things: {"id": 1, "method": "Target.createTarget", "params": {"url": "http://www.google.com"}} It does what you want, over a web socket interface rather than HTTP.
  - TicklishTiger 8y ago
    That looks interesting. Where does that ws tool come from? Is it this one? https://packages.debian.org/stretch/node-ws https://packages.debian.org/stretch/node-ws How do you send javascript commands over that protocol?
- deleted 8y ago
  [deleted]

androidgirl 8y ago

You can do this with Selenium pretty easily, and low level webdrivers support it too.

TicklishTiger 8y ago
I would give low level webdrivers a go. But so far I have not even figured out how to install one for Chromium on Debian. apt install chromedriver gives me: Package 'chromedriver' has no installation candidate There is something called "chromium-chromedriver". Let me try that ... one moment ... Ok. So I start it via: apt install chromium-chromedriver Now according to the docs, this should create a browser: curl -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }' http://localhost:9515/session Ha! It works! So chromedriver might be a solution!

SSchick 8y ago

What you are describing is exactly what puppeteer uses internally, you might want to explore their code base.

dlkinney 8y ago

While not currently "easy", there exists the Chrome Devtools Protocol.[0] I'm not aware of a CLI utility that communicates with it, but it wouldn't be impossible to make one that fulfills what you're looking for. A second tool could then act as a REST proxy, if calling the commands via curl is really your jam. I think you've given my weekend some purpose. Lemme see what I can pull together... [0] https://chromedevtools.github.io/devtools-protocol/ https://chromedevtools.github.io/devtools-protocol/

TicklishTiger 8y ago
Yes, that might work. Maybe an even better approach is to use chromium-chromedriver. I just got it working like this: apt install chromium-chromedriver chromedriver This seems to create a service that listens on port 9515 for standardized commands to remote control a chromium instance. The commands seem to be specified by the W3C: https://www.w3.org/TR/webdriver/ https://www.w3.org/TR/webdriver/ I got it to open a browser with this curl command: curl -d '{ "desiredCapabilities": { "caps": { "nativeEvents": false, "browserName": "chrome", "version": "", "platform": "ANY" } } }' http://localhost:9515/session I have not yet figured out how to send javascript commands though.

aasasd 8y ago

With ‘native messaging,’ you can have your program communicate with your extension, so the extension could then do everything that's available to it via the API. Won't be surprised if an extension like this already exists.

fake-name 8y ago

I've written something[1] that can basically do this, though it's non-interactive. The CLI interface is somewhat incomplete as-of-now, but it'd be fairly easy to add more comprehensive tools. https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController