4 ms·

Ranked #4 on HN at the moment and no comments. So I'll just say hi. (Selenium project creator here. I had nothing to do with this announcement, but feel free to

by hugs 2y ago

Ranked #4 on HN at the moment and no comments. So I'll just say hi. (Selenium project creator here. I had nothing to do with this announcement, but feel free to ask me anything!) My hot take on things: When the Puppeteer team left Google to join Microsoft and continue the project as Playwright, that left Google high and dry. I don't think Google truly realized how complementary a browser automation tool is to an AI-agent strategy. Similar to how they also fumbled the bag on transformer technology. (The T in GPT)... So Google had a choice, abandon Puppeteer and be dependent on MS/Playwright... or find a path forward for Puppeteer. WebDriver BiDi takes all the chocolatey goodness of the Chrome DevTools Protocol (CDP) that Puppeteer (and Playwright) are built on... and moves that forward in a standard way (building on the earlier success of the W3C WebDriver process that browser vendors and members of the Selenium project started years ago.) Great to see there's still a market for cross-industry standards and collaboration with this announcement from Mozilla today.

localfirst 2y ago
is it possible to now use Puppeteer from inside the browser? or do security concerns restrict this? what does Webdriver Bidi do and what do you mean by "taking the good stuff from CDP" I don't want to run my scrapes in the cloud and pay a monthly fee I want to run them locally. I want to run LLM locally too. I'm sick of SaaS
- hugs 2y ago
  Puppeteer controls a browser... from the outside... like a puppeteer controls a puppet. Other tools like Cypress (and ironically the very first version of Selenium 20 years ago) drive the browser from the inside using JavaScript. But we abandoned that "inside out" approach in later versions of Selenium because of the limitations imposed by the browser JS security sandbox. Cypress is still trying to make it work and I wish them luck. You could probably figure out how to connect Llama to Puppeteer. (If no one has done it, yet, that would be an awesome project.)
  - localfirst 2y ago
    I see im still looking for a way to control browser from the inside via an extension browser. very tough problem to solve.
    - fitsumbelay 2y ago
      I do alot quick manually scrapes via devtools you could try this Chrome web scraper extension - https://chromewebstore.google.com/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn https://chromewebstore.google.com/detail/web-scraper-free-we...
    - hugs 2y ago
      Yup. Lately, I've been doing it a completely different way (but still from the outside)... Using a Raspberry Pi as a fake keyboard and mouse. (Makes more sense in the context of mobile automation than desktop.) What's good for security is generally bad for automation... and trying to automate from inside a heavily secured sandbox is... frustrating. It works a little bit (as Cypress folks more recently learned), but you can never get to 100% covering all the things you'd want to cover. Driving from the outside is easier... but still not easy!
      - localfirst 2y ago
        interesting so you are emulating hardware inputs from RPi how is it reading whats on the screen? computer vision?
        hugs 2y ago
        Not to make this an ad for my project, but I'm starting to document it more here: https://valetnet.dev/ https://valetnet.dev/ The Raspberry Pi is configured to use the USB HID protocol to look and act like a mouse and keyboard when plugged into a phone. (Android and iOS now support mouse and keyboard inputs). For video, we have two models: - "Valet Link" uses an HDMI capture card (and a multi-port dongle) to pull the video signal directly from the phone if available. (This applies to all iPhones and high-end Samsung phones.) - "Valet Vision" which uses the Raspberry Pi V3 camera positioned 200mm above the phone to grab the video that way. Kinda crazy, but it works when HDMI output is not available. The whole thing is also enclosed in a black box so light from the environment doesn't affect the video capture. Then once we have an image, yes, you use whatever library you want to process and understand what's in the image. I currently use OpenCV and Tesseract (with Python). Could probably write a book about the lessons learned getting a "vision first" approach to automation working (as opposed to the lower-level Puppeteer/Playwright/Selenium/Appium way to do it.
- fitsumbelay 2y ago
  webdriver bidi info -https://www.youtube.com/watch?v=6oXic6dcn9w https://www.youtube.com/watch?v=6oXic6dcn9w local scraping howto - https://www.freecodecamp.org/news/web-scraping-in-javascript-with-puppeteer/ https://www.freecodecamp.org/news/web-scraping-in-javascript... local LLM framework - https://ollama.com/ https://ollama.com/
- jgraham 2y ago
  > Is it possible to now use Puppeteer from inside the browser? Talking about WebDriver (BiDi) in general rather than Puppeteer specifically, it depends what exactly you mean. Classic WebDriver is a HTTP-based protocol. WebDriver BiDi uses websockets (although other transports are a possibility for the future). Script running inside the browser can create HTTP connections and create websockets connections, so you can create a web page that implements a WebDriver or WebDriver BiDi client. But of course you need to have a browser to connect to, and that needs to be configured to actually allow connections from your host; for obvious security reasons that's not allowed by default. This sounds a bit obscure, but it can be useful. Firefox devtools is implemented in HTML+JS in the browser (like the rest of the Firefox UI), and can connect to a different Firefox instance (e.g. for debugging mobile Firefox from desktop). The default runner for web-platform-tests drives the browser from the outside (typically) using WebDriver, but it also provides an API so the in-browser tests can access some WebDriver commands.
- hoten 2y ago
  Yes. I'm not aware of any documentation walking one through it though. There is a extension api that exposes a CDP connection [1][2] You can create a Puppeteer.Browser given a CDP connection. You can bundle Puppeteer in a browser (we do this in Lighthouse/Chrome DevTools[3]). These two things is probably enough to get it working, though it may be limited to the active tab. [1] https://chromedevtools.github.io/devtools-protocol/#:~:text=DevTools%20protocol%20via%20Chrome%20extension https://chromedevtools.github.io/devtools-protocol/#:~:text=... [2] https://stackoverflow.com/a/55284340/24042444 https://stackoverflow.com/a/55284340/24042444 [3] https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/front_end/services/puppeteer/PuppeteerConnection.ts;l=79-98;drc=67e3886186dee80e5cb91c598a1552e1f5b0e65f https://source.chromium.org/chromium/chromium/src/+/main:thi...
SomaticPirate 2y ago
If I wanted to write some simple web-automation as a DevOps engineer with little javascript (or webdev experience at all) what tool would you recommend? Some example use cases would be writing some basic tests to validate a UI or automate some form-filling on a javascript based website with no API.
- hugs 2y ago
  Unironically, ask ChatGPT (or your favorite LLM) to create a hello world WebDriver or Puppeteer script (and installation instructions) and go from there.
  - righthand 2y ago
    “Go ask ChatGPT” is the new “RTFM”.
    - hugs 2y ago
      sorry, not sorry?
      - distortedsignal 2y ago
        I don't think they're criticizing - I think it's observation. It makes a lot of sense, and we're early-ish to the tech cycle. Reading the Manual/Google/ChatGPT are all just tools in the toolbelt. If you (an expert) is giving this advice, it should become mainstream soon-ish.
        0x1ch 2y ago
        I think this is where personal problem solving skills matter. I use ChatGPT to start off a lot of new ideas or projects with unfamiliar tools or libraries I will be using, however the result isn't always good. From here, a good developer will take the information from the A.I tool and look further into current documentation to supplement. If you can't distinguish bad from good with LLMs, you might as well be throwing crap at the wall hoping it will stick.
anothername12 2y ago
Is the WebDriver standard a good one? (Relative to playwright I guess) I seem to recall some pains implementing it a few years ago.
huy-nguyen 2y ago
What’s the relationship between Selenium, Puppeteer and Webdriver BiDi? I’m a happy user of Playwright. Is there any reason why I should consider Selenium or Puppeteer?
- imiric 2y ago
  > Is there any reason why I should consider Selenium or Puppeteer? I'm not a heavy user of these tools, but I've dabbled in this space. I think Playwright is far ahead as far as features and robustness go compared to alternatives. Firefox has been supported for a long time, as well as other features mentioned in this announcement like network interception and preload scripts. CDP in general is much more mature than WebDriver BiDi. Playwright also has a more modern API, with official bindings in several languages. One benefit of WebDriver BiDi is that it's in process of becoming a W3C standard, which might lead to wider adoption eventually. But today, I don't see a reason to use anything other than Playwright. Happy to read alternative opinions, though.
  - creesch 2y ago
    Both Selenium and Playwright are very solid tools, a lot simply comes down to choice and experience. One of the benefits of using Selenium is the extensive ecosystem surrounding it. Things like Selenium grid make parallel and cross-browser testing much easier either on self hosted hardware or through services like saucelabs. Playwright can be used with similar services like browserstack but AFAIK that requires an extra layer of their in-house SDK to actually make it work. Selenium also supports more browsers, although you can wonder how much use that is given the Chrome dominance these days. Another important difference is that Playwright really is a test automation framework, where Selenium is "just" a browser automation library. With Selenium you need to bring the assertion library, testrunner, reporting in yourself.
- Vinnl 2y ago
  I think Playwright depends on forking the browsers to support the features they need, so that may be less stable than using a standard explicitly supported by the browsers, and/or more representative of realistic browser use.
nox101 2y ago
Last time I tried playwright it required custom versions of the browsers. That meant it was impossible to use with any newer browser features. That made it impossible to use if you wanted to target new and advanced use cases or prep a site in expectation of some new API feature that just shipped or is expected to ship soon. If you used playwright, write tons of tests, then hear about some new browser feature you want to target to get ahead of your competition, you'd have to refactor all of your tests away from playwright to something that could target chrome canary or firefox nightly or safari technology preview. Has that changed?
- twic 2y ago
  It works for me with stock Chromium and Chrome on Linux. But for Firefox, i apparently need a custom patched build, which isn't available for the distro i run, so i haven't confirmed that.
- tracker1 2y ago
  IIRC, you can use the system installed browser, but need to know the executable path when launching. I remember it being a bit of a pain to do, but have done it.

by hugs 2y ago

localfirst 2y ago

is it possible to now use Puppeteer from inside the browser? or do security concerns restrict this? what does Webdriver Bidi do and what do you mean by "taking the good stuff from CDP" I don't want to run my scrapes in the cloud and pay a monthly fee I want to run them locally. I want to run LLM locally too. I'm sick of SaaS

hugs 2y ago
Puppeteer controls a browser... from the outside... like a puppeteer controls a puppet. Other tools like Cypress (and ironically the very first version of Selenium 20 years ago) drive the browser from the inside using JavaScript. But we abandoned that "inside out" approach in later versions of Selenium because of the limitations imposed by the browser JS security sandbox. Cypress is still trying to make it work and I wish them luck. You could probably figure out how to connect Llama to Puppeteer. (If no one has done it, yet, that would be an awesome project.)
- localfirst 2y ago
  I see im still looking for a way to control browser from the inside via an extension browser. very tough problem to solve.
  - fitsumbelay 2y ago
    I do alot quick manually scrapes via devtools you could try this Chrome web scraper extension - https://chromewebstore.google.com/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn https://chromewebstore.google.com/detail/web-scraper-free-we...
  - hugs 2y ago
    Yup. Lately, I've been doing it a completely different way (but still from the outside)... Using a Raspberry Pi as a fake keyboard and mouse. (Makes more sense in the context of mobile automation than desktop.) What's good for security is generally bad for automation... and trying to automate from inside a heavily secured sandbox is... frustrating. It works a little bit (as Cypress folks more recently learned), but you can never get to 100% covering all the things you'd want to cover. Driving from the outside is easier... but still not easy!
    - localfirst 2y ago
      interesting so you are emulating hardware inputs from RPi how is it reading whats on the screen? computer vision?
      - hugs 2y ago
        Not to make this an ad for my project, but I'm starting to document it more here: https://valetnet.dev/ https://valetnet.dev/ The Raspberry Pi is configured to use the USB HID protocol to look and act like a mouse and keyboard when plugged into a phone. (Android and iOS now support mouse and keyboard inputs). For video, we have two models: - "Valet Link" uses an HDMI capture card (and a multi-port dongle) to pull the video signal directly from the phone if available. (This applies to all iPhones and high-end Samsung phones.) - "Valet Vision" which uses the Raspberry Pi V3 camera positioned 200mm above the phone to grab the video that way. Kinda crazy, but it works when HDMI output is not available. The whole thing is also enclosed in a black box so light from the environment doesn't affect the video capture. Then once we have an image, yes, you use whatever library you want to process and understand what's in the image. I currently use OpenCV and Tesseract (with Python). Could probably write a book about the lessons learned getting a "vision first" approach to automation working (as opposed to the lower-level Puppeteer/Playwright/Selenium/Appium way to do it.
fitsumbelay 2y ago
webdriver bidi info -https://www.youtube.com/watch?v=6oXic6dcn9w https://www.youtube.com/watch?v=6oXic6dcn9w local scraping howto - https://www.freecodecamp.org/news/web-scraping-in-javascript-with-puppeteer/ https://www.freecodecamp.org/news/web-scraping-in-javascript... local LLM framework - https://ollama.com/ https://ollama.com/
jgraham 2y ago
> Is it possible to now use Puppeteer from inside the browser? Talking about WebDriver (BiDi) in general rather than Puppeteer specifically, it depends what exactly you mean. Classic WebDriver is a HTTP-based protocol. WebDriver BiDi uses websockets (although other transports are a possibility for the future). Script running inside the browser can create HTTP connections and create websockets connections, so you can create a web page that implements a WebDriver or WebDriver BiDi client. But of course you need to have a browser to connect to, and that needs to be configured to actually allow connections from your host; for obvious security reasons that's not allowed by default. This sounds a bit obscure, but it can be useful. Firefox devtools is implemented in HTML+JS in the browser (like the rest of the Firefox UI), and can connect to a different Firefox instance (e.g. for debugging mobile Firefox from desktop). The default runner for web-platform-tests drives the browser from the outside (typically) using WebDriver, but it also provides an API so the in-browser tests can access some WebDriver commands.
hoten 2y ago
Yes. I'm not aware of any documentation walking one through it though. There is a extension api that exposes a CDP connection [1][2] You can create a Puppeteer.Browser given a CDP connection. You can bundle Puppeteer in a browser (we do this in Lighthouse/Chrome DevTools[3]). These two things is probably enough to get it working, though it may be limited to the active tab. [1] https://chromedevtools.github.io/devtools-protocol/#:~:text=DevTools%20protocol%20via%20Chrome%20extension https://chromedevtools.github.io/devtools-protocol/#:~:text=... [2] https://stackoverflow.com/a/55284340/24042444 https://stackoverflow.com/a/55284340/24042444 [3] https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/front_end/services/puppeteer/PuppeteerConnection.ts;l=79-98;drc=67e3886186dee80e5cb91c598a1552e1f5b0e65f https://source.chromium.org/chromium/chromium/src/+/main:thi...

SomaticPirate 2y ago

If I wanted to write some simple web-automation as a DevOps engineer with little javascript (or webdev experience at all) what tool would you recommend? Some example use cases would be writing some basic tests to validate a UI or automate some form-filling on a javascript based website with no API.

hugs 2y ago
Unironically, ask ChatGPT (or your favorite LLM) to create a hello world WebDriver or Puppeteer script (and installation instructions) and go from there.
- righthand 2y ago
  “Go ask ChatGPT” is the new “RTFM”.
  - hugs 2y ago
    sorry, not sorry?
    - distortedsignal 2y ago
      I don't think they're criticizing - I think it's observation. It makes a lot of sense, and we're early-ish to the tech cycle. Reading the Manual/Google/ChatGPT are all just tools in the toolbelt. If you (an expert) is giving this advice, it should become mainstream soon-ish.
      - 0x1ch 2y ago
        I think this is where personal problem solving skills matter. I use ChatGPT to start off a lot of new ideas or projects with unfamiliar tools or libraries I will be using, however the result isn't always good. From here, a good developer will take the information from the A.I tool and look further into current documentation to supplement. If you can't distinguish bad from good with LLMs, you might as well be throwing crap at the wall hoping it will stick.

anothername12 2y ago

Is the WebDriver standard a good one? (Relative to playwright I guess) I seem to recall some pains implementing it a few years ago.

huy-nguyen 2y ago

What’s the relationship between Selenium, Puppeteer and Webdriver BiDi? I’m a happy user of Playwright. Is there any reason why I should consider Selenium or Puppeteer?

imiric 2y ago
> Is there any reason why I should consider Selenium or Puppeteer? I'm not a heavy user of these tools, but I've dabbled in this space. I think Playwright is far ahead as far as features and robustness go compared to alternatives. Firefox has been supported for a long time, as well as other features mentioned in this announcement like network interception and preload scripts. CDP in general is much more mature than WebDriver BiDi. Playwright also has a more modern API, with official bindings in several languages. One benefit of WebDriver BiDi is that it's in process of becoming a W3C standard, which might lead to wider adoption eventually. But today, I don't see a reason to use anything other than Playwright. Happy to read alternative opinions, though.
- creesch 2y ago
  Both Selenium and Playwright are very solid tools, a lot simply comes down to choice and experience. One of the benefits of using Selenium is the extensive ecosystem surrounding it. Things like Selenium grid make parallel and cross-browser testing much easier either on self hosted hardware or through services like saucelabs. Playwright can be used with similar services like browserstack but AFAIK that requires an extra layer of their in-house SDK to actually make it work. Selenium also supports more browsers, although you can wonder how much use that is given the Chrome dominance these days. Another important difference is that Playwright really is a test automation framework, where Selenium is "just" a browser automation library. With Selenium you need to bring the assertion library, testrunner, reporting in yourself.
Vinnl 2y ago
I think Playwright depends on forking the browsers to support the features they need, so that may be less stable than using a standard explicitly supported by the browsers, and/or more representative of realistic browser use.

nox101 2y ago

Last time I tried playwright it required custom versions of the browsers. That meant it was impossible to use with any newer browser features. That made it impossible to use if you wanted to target new and advanced use cases or prep a site in expectation of some new API feature that just shipped or is expected to ship soon. If you used playwright, write tons of tests, then hear about some new browser feature you want to target to get ahead of your competition, you'd have to refactor all of your tests away from playwright to something that could target chrome canary or firefox nightly or safari technology preview. Has that changed?

twic 2y ago
It works for me with stock Chromium and Chrome on Linux. But for Firefox, i apparently need a custom patched build, which isn't available for the distro i run, so i haven't confirmed that.
tracker1 2y ago
IIRC, you can use the system installed browser, but need to know the executable path when launching. I remember it being a bit of a pain to do, but have done it.