4 ms·

is it possible to now use Puppeteer from inside the browser? or do security concerns restrict this? what does Webdriver Bidi do and what do you mean by "taking

is it possible to now use Puppeteer from inside the browser? or do security concerns restrict this? what does Webdriver Bidi do and what do you mean by "taking the good stuff from CDP" I don't want to run my scrapes in the cloud and pay a monthly fee I want to run them locally. I want to run LLM locally too. I'm sick of SaaS

hugs 2y ago
Puppeteer controls a browser... from the outside... like a puppeteer controls a puppet. Other tools like Cypress (and ironically the very first version of Selenium 20 years ago) drive the browser from the inside using JavaScript. But we abandoned that "inside out" approach in later versions of Selenium because of the limitations imposed by the browser JS security sandbox. Cypress is still trying to make it work and I wish them luck. You could probably figure out how to connect Llama to Puppeteer. (If no one has done it, yet, that would be an awesome project.)
- localfirst 2y ago
  I see im still looking for a way to control browser from the inside via an extension browser. very tough problem to solve.
  - fitsumbelay 2y ago
    I do alot quick manually scrapes via devtools you could try this Chrome web scraper extension - https://chromewebstore.google.com/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn https://chromewebstore.google.com/detail/web-scraper-free-we...
  - hugs 2y ago
    Yup. Lately, I've been doing it a completely different way (but still from the outside)... Using a Raspberry Pi as a fake keyboard and mouse. (Makes more sense in the context of mobile automation than desktop.) What's good for security is generally bad for automation... and trying to automate from inside a heavily secured sandbox is... frustrating. It works a little bit (as Cypress folks more recently learned), but you can never get to 100% covering all the things you'd want to cover. Driving from the outside is easier... but still not easy!
    - localfirst 2y ago
      interesting so you are emulating hardware inputs from RPi how is it reading whats on the screen? computer vision?
      - hugs 2y ago
        Not to make this an ad for my project, but I'm starting to document it more here: https://valetnet.dev/ https://valetnet.dev/ The Raspberry Pi is configured to use the USB HID protocol to look and act like a mouse and keyboard when plugged into a phone. (Android and iOS now support mouse and keyboard inputs). For video, we have two models: - "Valet Link" uses an HDMI capture card (and a multi-port dongle) to pull the video signal directly from the phone if available. (This applies to all iPhones and high-end Samsung phones.) - "Valet Vision" which uses the Raspberry Pi V3 camera positioned 200mm above the phone to grab the video that way. Kinda crazy, but it works when HDMI output is not available. The whole thing is also enclosed in a black box so light from the environment doesn't affect the video capture. Then once we have an image, yes, you use whatever library you want to process and understand what's in the image. I currently use OpenCV and Tesseract (with Python). Could probably write a book about the lessons learned getting a "vision first" approach to automation working (as opposed to the lower-level Puppeteer/Playwright/Selenium/Appium way to do it.
        localfirst 2y ago
        > Could probably write a book about the lessons learned getting a "vision first" approach to automation working ha that would be splendid! please do maybe even a blog on valetnet.dev (lovely site btw a demo or video would be a nice) I'm convinced vision first is the way to go despite people saying its slow the benefits are tremendous as lot of websites simply do not play nice with HTML and I do not like having to inspect XHR to figure out APIs SikuliX was my last love affair with this approach but eventually I lost interest in scraping and automation so I'm pleased to see people still working on vision first automation approaches.
        hugs 2y ago
  - weaksauce 2y ago
    are you using native messaging? there's a way to bridge a program running with full permissions inside the computer that could use puppeteer or the like. https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web... seems like it wouldn't be that hard to sync the two but the devil is in the details. also installing the native script is outside the purview of the webext so you need to have an installer.
    - mst 2y ago
      If it's a single file you could just make it a download. There's also the newer file system APIs (though in Safari you'll be missing features and need to put some things in a Web Worker).
  - namuorg 2y ago
    I do this for https://browserflow.app https://browserflow.app (and the AI version in development at https://browserbot.ai https://browserbot.ai) via the chrome.debugger API: https://developer.chrome.com/docs/extensions/reference/api/debugger https://developer.chrome.com/docs/extensions/reference/api/d...
fitsumbelay 2y ago
webdriver bidi info -https://www.youtube.com/watch?v=6oXic6dcn9w https://www.youtube.com/watch?v=6oXic6dcn9w local scraping howto - https://www.freecodecamp.org/news/web-scraping-in-javascript-with-puppeteer/ https://www.freecodecamp.org/news/web-scraping-in-javascript... local LLM framework - https://ollama.com/ https://ollama.com/
jgraham 2y ago
> Is it possible to now use Puppeteer from inside the browser? Talking about WebDriver (BiDi) in general rather than Puppeteer specifically, it depends what exactly you mean. Classic WebDriver is a HTTP-based protocol. WebDriver BiDi uses websockets (although other transports are a possibility for the future). Script running inside the browser can create HTTP connections and create websockets connections, so you can create a web page that implements a WebDriver or WebDriver BiDi client. But of course you need to have a browser to connect to, and that needs to be configured to actually allow connections from your host; for obvious security reasons that's not allowed by default. This sounds a bit obscure, but it can be useful. Firefox devtools is implemented in HTML+JS in the browser (like the rest of the Firefox UI), and can connect to a different Firefox instance (e.g. for debugging mobile Firefox from desktop). The default runner for web-platform-tests drives the browser from the outside (typically) using WebDriver, but it also provides an API so the in-browser tests can access some WebDriver commands.
hoten 2y ago
Yes. I'm not aware of any documentation walking one through it though. There is a extension api that exposes a CDP connection [1][2] You can create a Puppeteer.Browser given a CDP connection. You can bundle Puppeteer in a browser (we do this in Lighthouse/Chrome DevTools[3]). These two things is probably enough to get it working, though it may be limited to the active tab. [1] https://chromedevtools.github.io/devtools-protocol/#:~:text=DevTools%20protocol%20via%20Chrome%20extension https://chromedevtools.github.io/devtools-protocol/#:~:text=... [2] https://stackoverflow.com/a/55284340/24042444 https://stackoverflow.com/a/55284340/24042444 [3] https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/front_end/services/puppeteer/PuppeteerConnection.ts;l=79-98;drc=67e3886186dee80e5cb91c598a1552e1f5b0e65f https://source.chromium.org/chromium/chromium/src/+/main:thi...

hugs 2y ago

Puppeteer controls a browser... from the outside... like a puppeteer controls a puppet. Other tools like Cypress (and ironically the very first version of Selenium 20 years ago) drive the browser from the inside using JavaScript. But we abandoned that "inside out" approach in later versions of Selenium because of the limitations imposed by the browser JS security sandbox. Cypress is still trying to make it work and I wish them luck. You could probably figure out how to connect Llama to Puppeteer. (If no one has done it, yet, that would be an awesome project.)

localfirst 2y ago
I see im still looking for a way to control browser from the inside via an extension browser. very tough problem to solve.
- fitsumbelay 2y ago
  I do alot quick manually scrapes via devtools you could try this Chrome web scraper extension - https://chromewebstore.google.com/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn https://chromewebstore.google.com/detail/web-scraper-free-we...
- hugs 2y ago
  Yup. Lately, I've been doing it a completely different way (but still from the outside)... Using a Raspberry Pi as a fake keyboard and mouse. (Makes more sense in the context of mobile automation than desktop.) What's good for security is generally bad for automation... and trying to automate from inside a heavily secured sandbox is... frustrating. It works a little bit (as Cypress folks more recently learned), but you can never get to 100% covering all the things you'd want to cover. Driving from the outside is easier... but still not easy!
  - localfirst 2y ago
    interesting so you are emulating hardware inputs from RPi how is it reading whats on the screen? computer vision?
    - hugs 2y ago
      Not to make this an ad for my project, but I'm starting to document it more here: https://valetnet.dev/ https://valetnet.dev/ The Raspberry Pi is configured to use the USB HID protocol to look and act like a mouse and keyboard when plugged into a phone. (Android and iOS now support mouse and keyboard inputs). For video, we have two models: - "Valet Link" uses an HDMI capture card (and a multi-port dongle) to pull the video signal directly from the phone if available. (This applies to all iPhones and high-end Samsung phones.) - "Valet Vision" which uses the Raspberry Pi V3 camera positioned 200mm above the phone to grab the video that way. Kinda crazy, but it works when HDMI output is not available. The whole thing is also enclosed in a black box so light from the environment doesn't affect the video capture. Then once we have an image, yes, you use whatever library you want to process and understand what's in the image. I currently use OpenCV and Tesseract (with Python). Could probably write a book about the lessons learned getting a "vision first" approach to automation working (as opposed to the lower-level Puppeteer/Playwright/Selenium/Appium way to do it.
      - localfirst 2y ago
        > Could probably write a book about the lessons learned getting a "vision first" approach to automation working ha that would be splendid! please do maybe even a blog on valetnet.dev (lovely site btw a demo or video would be a nice) I'm convinced vision first is the way to go despite people saying its slow the benefits are tremendous as lot of websites simply do not play nice with HTML and I do not like having to inspect XHR to figure out APIs SikuliX was my last love affair with this approach but eventually I lost interest in scraping and automation so I'm pleased to see people still working on vision first automation approaches.
        hugs 2y ago
- weaksauce 2y ago
  are you using native messaging? there's a way to bridge a program running with full permissions inside the computer that could use puppeteer or the like. https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web... seems like it wouldn't be that hard to sync the two but the devil is in the details. also installing the native script is outside the purview of the webext so you need to have an installer.
  - mst 2y ago
    If it's a single file you could just make it a download. There's also the newer file system APIs (though in Safari you'll be missing features and need to put some things in a Web Worker).
- namuorg 2y ago
  I do this for https://browserflow.app https://browserflow.app (and the AI version in development at https://browserbot.ai https://browserbot.ai) via the chrome.debugger API: https://developer.chrome.com/docs/extensions/reference/api/debugger https://developer.chrome.com/docs/extensions/reference/api/d...

fitsumbelay 2y ago

webdriver bidi info -https://www.youtube.com/watch?v=6oXic6dcn9w https://www.youtube.com/watch?v=6oXic6dcn9w local scraping howto - https://www.freecodecamp.org/news/web-scraping-in-javascript-with-puppeteer/ https://www.freecodecamp.org/news/web-scraping-in-javascript... local LLM framework - https://ollama.com/ https://ollama.com/

jgraham 2y ago

> Is it possible to now use Puppeteer from inside the browser? Talking about WebDriver (BiDi) in general rather than Puppeteer specifically, it depends what exactly you mean. Classic WebDriver is a HTTP-based protocol. WebDriver BiDi uses websockets (although other transports are a possibility for the future). Script running inside the browser can create HTTP connections and create websockets connections, so you can create a web page that implements a WebDriver or WebDriver BiDi client. But of course you need to have a browser to connect to, and that needs to be configured to actually allow connections from your host; for obvious security reasons that's not allowed by default. This sounds a bit obscure, but it can be useful. Firefox devtools is implemented in HTML+JS in the browser (like the rest of the Firefox UI), and can connect to a different Firefox instance (e.g. for debugging mobile Firefox from desktop). The default runner for web-platform-tests drives the browser from the outside (typically) using WebDriver, but it also provides an API so the in-browser tests can access some WebDriver commands.

hoten 2y ago

Yes. I'm not aware of any documentation walking one through it though. There is a extension api that exposes a CDP connection [1][2] You can create a Puppeteer.Browser given a CDP connection. You can bundle Puppeteer in a browser (we do this in Lighthouse/Chrome DevTools[3]). These two things is probably enough to get it working, though it may be limited to the active tab. [1] https://chromedevtools.github.io/devtools-protocol/#:~:text=DevTools%20protocol%20via%20Chrome%20extension https://chromedevtools.github.io/devtools-protocol/#:~:text=... [2] https://stackoverflow.com/a/55284340/24042444 https://stackoverflow.com/a/55284340/24042444 [3] https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/front_end/services/puppeteer/PuppeteerConnection.ts;l=79-98;drc=67e3886186dee80e5cb91c598a1552e1f5b0e65f https://source.chromium.org/chromium/chromium/src/+/main:thi...