24 ms·
Prototype: Puppeteer for Firefox
- eknkc 7y agoIt is kind of usable but is still missing stuff. You can check the status here: https://aslushnikov.github.io/ispuppeteerfirefoxready/ https://aslushnikov.github.io/ispuppeteerfirefoxready/
- SideburnsOfDoom 7y agoIn case you're wondering, and the linked page doesn't explain: > Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium. https://developers.google.com/web/tools/puppeteer/ https://developers.google.com/web/tools/puppeteer/
- Gipetto 7y agoThank you. This industry has a huge failure mode in touting tech used in projects/frameworks without linking out to at least the source so that someone can learn about them. Can't tell you how many things I've bailed on because it was a pile of obscure library references that weren't (to me) worth looking up.
- couchand 7y agoTo be fair, in this case the linked page is a subdirectory of the main Puppeteer project.
- kaycebasques 7y agoThis is also an experimental project. Who here thoroughly documents their experiments? Although to be fair we did announce it at I/O so that calls for more documentation. Disclosure: Chrome DevTools docs guy
- SideburnsOfDoom 7y agoIndeed, there is a skill when doing technical writing of putting yourself in the position of a reader, particularly one who is competent but who doesn't already know the thing that you are trying to explain, or the context. It's not as common as it should be.
- godelski 7y agoI always find it weird that people constantly try to "sell" me (us) something but never tell me what that thing is or why I want it.
- SideburnsOfDoom 7y agoI can only assume that anyone who submits "Puppeteer for Firefox" or "x for y" to Hackernews, is someone who works daily with Puppeteer / x and has the necessary context and doesn't notice that it's not aimed at a more general audience (1). Then it gets upvoted to a more general audience, who ask "Puppeteer? what's that?" 1) https://news.ycombinator.com/item?id=20507728 https://news.ycombinator.com/item?id=20507728
- fsiefken 7y agois there a benefit to using firefox with puppeteer compared to chromium and puppeteer? or is it just a matter of choice and being able to also use firefox for automated testing?
- bjhkx 7y agoI guess it's a matter of choice for people who still use Firefox.
- brna 7y ago*people who still use Chrome.
- bjhkx 7y agoUh? http://gs.statcounter.com/browser-market-share/desktop/worldwide/#monthly-201506-201906 http://gs.statcounter.com/browser-market-share/desktop/world...
- trishmapow2 7y agoInteresting, I thought Firefox usage would be increasing based on all the hype/people converting on HN/other tech websites but the market share is actually falling. Guess it puts into perspective the size of that cohort compared to the rest of the world.
- Crinus 7y agoAfter Firefox broke most extensions with their removal of XUL extensions and embrace of "WebExtensions" (aka Chrome extensions), it lost a major differentiation point from Chrome. People simply have no (immediate, practical) reason to use Firefox. Back in the Mozilla days i was using Mozilla because it was very powerful and flexible despite being very slow (i remember watching Mozilla 0.6's dialog boxes draw themselves) and many pages didn't work with it because everyone only cared about IE (...and i'd put the RIIR crowd to shame with my "evangelization" :-P). Nowadays i use Firefox mainly out of inertia. Perhaps if Google disables most of the stuff adblockers rely on, there will be again a reason to use Firefox.
- sbmthakur 7y agoWhat is the equivalent of Chrome Dev tools protocol here?
- world32 7y agoIt says its based on the mozilla remote protocol: https://wiki.mozilla.org/Remote https://wiki.mozilla.org/Remote "The Firefox Remote Protocol is a low-level debugging interface based on the CDP protocol."
- deleted 7y ago[deleted]
- world32 7y agoNice work, I have been using puppeteer with chrome recently and it is a great piece of kit. If the prototype is successful will you keep the chrome and firefox packages separate or merge them and have an abstraction layer so puppeteer can issue commands regardless of chrome/firefox being used for the browser?
- felixfbecker 7y agoThey already have the same API
- world32 7y agoYeah, that doesn't answer my question though ;)
- pawelmurias 7y agoMerging them in a single package gives you nothing at a significant price. puppeteer = wantFox ? require('puppeteer-firefox') : require('puppeteer')
- world32 7y agoSure but now there are two separate codebases for the same API. A lot of code will be duplicated between the two codebases, you'll have to maintain two sets of issues on github etc. If it were me I would aim to merge the firefox puppeteer into chrome puppeteer but fair enough if they want to keep them separate. EDIT: I guess it depends on how much of the code in the puppeteer chrome codebase is tightly coupled to chrome dev tools. Like if 25% of specific to chrome, then that would mean that 75% of the code can be shared between the chrome implementation and the firefox one, in that case I think merging and hiding the 25% chrome/firefox specific stuff behind an abstraction layer is the best option.
- zserge 7y agoHuh. That's great news. Time to add Firefox support to Lorca lib, I guess.
- sebazzz 7y agoFor automated testing, why Puppeteer instead of Selenium?
- d33 7y agoMuch better access to low-level information. You can hook (and mock) requests, see what POST data was sent and basically do lots of magic. Also, my experience with Selenium under Python is that doing anything bigger quickly turns into one giant timing hack.
- rat87 7y agoback when I did selenium (with python) I wait-ed for the wait for next page to load (html element becomes stale) and for textbox/checkbox/etc. you are selecting to be present with a few tiny helper functions. That took most of the magic sleeps out although I still needed timeouts on the waits
- HunOL 7y agoIf it really E2E from user perspective you shouldn't doing any magic and manipulate POST data.
- danShumway 7y agoYou're not technically wrong, but unit/integration/E2E are not hard categories for tests, they're just points along a continuum. Sometimes I want to directly control a browser to run a test or examine a UI component, but I still want to mock parts of my application. This is one of the biggest weaknesses with Selenium -- it takes a very narrow view of what testing is, so it becomes much less useful as soon as you're trying to do anything interesting. Different applications call for different testing strategies. Selenium could be a low-level browser controller that allows you to write strict E2E tests, but instead it's a low-level browser controller that is almost fanatical about only supporting strict E2E tests, even when it means that basic actions need to be more complicated. I think that's a design mistake, but whatever.
- rehemiau 7y agoHow is the DevTools/Remote protocol better (or worse) than WebDriver prptocol for integration testing?
- nicoburns 7y agoIt's much more reliable.
- Congeec 7y agoHow? With selenium I have a case that WebElement.click() fails but it doesn’t throw. Also Select(...).select_by_visible_text() throws WebDriverException, StaleElementException, etc even if you WebDriverWait()’d for it to be available. To me these show selenium is unreliable. How puppeteer is reliable pertaining to these cases?
- zelo 7y agoIt's probably because in the meantime dom nodes has been replaced by new ones. Selenium has it's quirks but when you understand how it works and it's limitations and user centric philosophy there is nothing that can stop you from writing rock solid stable tests.
- Congeec 7y agoSorry I didn't defined the problems clearly and thank your for your help. I am using selenium using python for web automation, not testing. The latter problem was by and large solved by subclassing WebDriverWait by adding common Exceptions related to nodes staleness. The former problem was tackled by writing a for loop detecting changes of the context to know if WebElement.click() fails silently. The thing is I have to write these workarounds everywhere. What's more, I inject javascript code a lot to avoid nodes staleness caused by network latency. I believe selenium is not designed for writing lots of javascript. And javascript code in a string of python can not be linted, which increases debugging workload. To this point, the cost of these hacks is too much in my project. I find puppeteer play nicely with javascript after trying. But the difference and reliability between waitFor() in puppeteer and WebDriverWait() in selenium still remain questionable. I am wondering if using puppeteer can ease pains mentioned above.
- ldng 7y agoI was under the impression it was about to be retired ? Did something change ?
- thatwasunusual 7y agoThis is nice work. Does anyone know if it's possible to enable reader view "programatically", though?
- fuzzy2 7y agoSure, just go to `about:reader?url=whatever` Or did you mean something else?
- thatwasunusual 7y agoThis is exactly what I'm after. Thanks a lot! :)
- blodovnik 7y agoFirefox has lost its lead because it's lost the developers. Chrome has invested massively in developer tools for chrome and it's shows.... I think chrome is the best software ever written. The developer tools are so powerful that it'd hard to even know the full breadth of what's in there. Firefox offers nothing to developers over chrome, which is sad.
- daveFNbuck 7y ago> Firefox offers nothing to developers over chrome, which is sad. Firefox has multi-account containers. Having different tabs logged in to different accounts is huge for me.
- Bjartr 7y agoI use Chrome profiles to great effect.
- daveFNbuck 7y agoCan't you only have one profile open at a time, and each one is linked to a different Google account? Firefox allows you to have each tab open in a different container, and they don't have to be tied to anything in particular. I can have different tabs open to the AWS console for each AWS account I use. I can also switch tabs to change between different user accounts to test interactions between users on the site I'm developing.
- jamesgeck0 7y ago> Can't you only have one profile open at a time No. Each profile opens in a new Chrome window. You can have as many open as you'd like. > and each one is linked to a different Google account? No. Local profiles have always been a thing. Chrome's implementation has better UX for privacy IMHO; it's harder to accidentally open a tab in a different profile than expected, which I do _all the time_ in Firefox. In Chrome, Cmd+T opens a new tab in the active profile. In Firefox, Cmd+T opens a new tab in the default profile.
- felixfbecker 7y agoWe're using it to test our browser extension at Sourcegraph in both Firefox and Chrome: https://github.com/sourcegraph/sourcegraph/blob/master/browser/src/e2e/e2e.test.ts https://github.com/sourcegraph/sourcegraph/blob/master/brows...
- kyriakos 7y agoHow's pdf rendering in Firefox? Chrome headless has its issues with headers and footers
- mjepronk 7y agoIt's good, however it's not possible to use the remote debugging protocol for printing to PDF yet.
- deleted 7y ago[deleted]
- gryffin 7y agoFirefox uses PDF.js[1] to render pdf. This is a standalone project to render pdf in the browser using html and js. If I remember correctly, you can even render pdf in chrome using this. [1]https://mozilla.github.io/pdf.js/ https://mozilla.github.io/pdf.js/
- tmaly 7y agoI just used Chrome headless to render several thousand pdfs last week and I saw no issues in version 73
- mkl 7y agoDid you have a reason to not use PDFium directly?
- dmhimanshu 7y agogreat work on firefox on github.com nice one require attention by all developers on github check it out
- dxxvi 7y agoFor what I see, puppeteer is very much like testcafe (able to run with headless/full Firefox, Chrome, Edge ..., but testcafe doesn't need a custom-built Firefox). TestCafe can run javascript code on the page and return the data back, that means it can be used to get data in POST requests. It can take screenshots too. If anybody is familiar with both tools, could you please what this tool can do but the other cannot?
- nostalgk 7y agoI have never heard of TestCafe, but I've used Puppeteer for a variety of projects with Chrome. My latest one was using it to convert HTML files into PDFs en masse.
- mikepurvis 7y agoIs there an advantage to that approach over the more direct wkhtmltopdf?
- royjacobs 7y agoIt looks exactly like it would in the latest version of Chrome and is not dependent on Qt's Webkit version.
- nostalgk 7y agoThis approach, for us, proved fastest when we scaled it using Kubernetes compared to the others we tried.
- jamesgeck0 7y agoMy experience with wkhtmltopdf a few years ago was that the PDFs that it rendered could look quite different from the PDFs that Chrome or Firefox would render.
- Vinnl 7y agoAs far as I know, Puppeteer actually controls the browser, whereas TestCafe injects things into the page - risking influencing the results of your tests. I've never used TestCafe, so I don't know how justified that worry is.
- breatheoften 7y agoPuppeteer is really really good. I just discovered it and used it for something and it’s a blast to work with. I’m seriously thinking there is a cloud api play in the near future — a puppeteer Page is a pretty awesome container format ... a Puppeteer Page or BrowserContext per request would provide an awesome cloud-function-like programming model I think...
- anandchowdhary 7y agoI really like https://microlink.io https://microlink.io for browser-based automation.
- digitarald 7y agoFor anybody who wants to follow along, the Firefox work is tracked here: https://gitter.im/webhintio/Firefox https://gitter.im/webhintio/Firefox The experiment was a great start to see what is needed to support Puppeteer. The next step is coming up with a well-integrated architecture. Find me on the web to talk to me about your use cases and ideas for Puppeteer.
- digitarald 7y agoActual link: https://bugzilla.mozilla.org/show_bug.cgi?id=1545057 https://bugzilla.mozilla.org/show_bug.cgi?id=1545057
- nealrs 7y agoTangentially related => I used ffmpeg + puppeteer to build StoryScroll https://neal.rs/app https://neal.rs/app, which turns blog posts into scrolling videos for social media. I also used to build a desktop app that my team uses to download Alexa Flash Briefing metrics (because there's no API for that) Puppeteer's full-screen screenshots & DOM manipulation abilities are clutch!!
- no_wizard 7y agoLooks like they’re quite far along in developing this according to this status page https://aslushnikov.github.io/ispuppeteerfirefoxready/ https://aslushnikov.github.io/ispuppeteerfirefoxready/
- deleted 7y ago[deleted]
- marmada 7y agoHas anyone else found Puppeteer to be very slow? I was using it to scrape a webpage and found it to be much slower than Selenium.
- floatingatoll 7y agoWhat is the use case you're trying out Puppeteer for?
- marmada 7y agoScraping all the posts from a Facebook group.
- mintplant 7y agoHuh. I worked on exactly this idea as a Mozilla intern in 2017. I wrote a collection of Rust crates (mentioned in [0]) comprising auto-generated types for all Chrome DevTools Protocol messages [1], (de)serialization, and a server lib to handle both the initial HTTP handshake and subsequent command execution over WebSockets. I used that to build out an initial CDP server inside Firefox (and Servo) with support for a couple of sample commands. At the end of the internship I handed over everything I'd built by that point, with the idea that they (Mozilla) would turn it into a finished project. But I never saw any activity on that front after I left, so I assumed the idea'd been scrapped. Feels very weird seeing this pop up out of the blue on HN today. [0] https://bugzilla.mozilla.org/show_bug.cgi?id=1523104#c1 https://bugzilla.mozilla.org/show_bug.cgi?id=1523104#c1 [1] Complete with documentation comment generation; I found the result of running rustdoc over that crate super useful as an alternative to the official CDP docs site: https://www.spinda.net/files/mozilla/rust-cdp/doc/cdp/index.html https://www.spinda.net/files/mozilla/rust-cdp/doc/cdp/index....
- itwy 7y agoAre you saying they didn't credit you for your work?
- mintplant 7y agoI'm not saying that. Just sharing my experience and expressing surprise at the reappearance of what I thought was a dead project.
- kaycebasques 7y agoThere’s a lot of debate between the browser vendors and automation tools around whether the Chrome DevTools Protocol is the right abstraction layer to focus browser automation around. It’s a nuanced topic and if you dig into it you’ll see that all the different sides make good arguments. My guess is that this debate stalled your project. I’m putting myself out here on the topic because having a coherent and enjoyable cross-browser integration testing story would be a huge win for the web at large and it’s something worth making people aware of in the hopes that we can find common ground. Edit: I actually have no right to comment on what stalled your project and have no idea what happened there. I just wanted to discuss the fact that we need a good abstraction layer but seem stuck in debate. Disclosure: Chrome DevTools docs guy, just speaking personally based on my research into the topic over the last year
- kaycebasques 7y agoWe announced this at Google I/O 2019 and mentioned it in What’s New In DevTools (Chrome 76): https://developers.google.com/web/updates/2019/05/devtools#puppeteer https://developers.google.com/web/updates/2019/05/devtools#p...
- tnolet 7y agoAnyone who wants to play around with some Puppeteer examples, I built https://puppeteersandbox.com https://puppeteersandbox.com. If you want to record your own scripts, I also am actively developing https://checklyhq.com/Puppeteer-recorder https://checklyhq.com/Puppeteer-recorder, a Chrome extension.
- folkhack 7y agoThat second link is a 404, although I love your 404 page GIF =) Checkly looks like a great product!
- tnolet 7y agoThanks, building a SaaS is 99% getting the 404 pages right!
- pastage 7y agoYou misspelled the URL https://checklyhq.com/puppeteer-recorder https://checklyhq.com/puppeteer-recorder Love recording macros
- tnolet 7y agoArgh, thanks for the heads up
- pilaf 7y agoWeird that they'd pick a red panda for the logo.