20 ms·
Theheadless.dev – open source Puppeteer and Playwright knowledge base
- skywal_l 6y agoChrome only...
- rozenmd 6y ago"Playwright provides a set of APIs to automate Chromium, Firefox and WebKit browsers."
- skywal_l 6y agoIndeed I missed that. So playwright is definitely more interesting. What is surprising is the linked site is quoting the FAQ of playwright that has been removed a couple of months ago: https://github.com/microsoft/playwright/pull/1930/files https://github.com/microsoft/playwright/pull/1930/files That FAQ seems to be giving a lot of perspective on the subject. Wondered why it was removed.
- tudorconstantin 6y agoFor puppetteer, but playwright also works with Firefox and webkit
- tnolet 6y agoOP here. Puppeteer is actually starting to support Firefox also quite extensively. But Playwright takes the crown for now.
- tudorconstantin 6y agoQuick question: if you were to start a new project today, what would be your choice between puppeteer and playwright? Haven't worked extensively with them, but it seems to me playwright is the no-brainer now because of the guys behind it (the creators of puppeteer), their experience and lessons learned with puppeteer and also the support for Firefox and webkit.
- tnolet 6y agoI guess so, the emergence of Playwright is therefore really interesting in general. It's two frameworks by arguably the same folks. Time will tell what will happen. Competition is never bad!
- simlevesque 6y agoBeware, when I tested Playwright to migrate from Puppeteer, many features were lacking or less polished.
- ravenstine 6y agoHow long ago? I've had nothing but a good experience with Playwright.
- arjun27 6y agoThanks for the feedback. I'm from the Playwright team, and I'd love to know more on what was missing for you.
- vulpesx2 6y agoI've never really worked too heavily with headless, so what would be some examples of 'real-life' applications using this API? Looking for some inspiration to maybe build a side project around this :)
- tnolet 6y agoObviously tooting my own horn here, but these are some products: - https://checklyhq.com https://checklyhq.com - synthetic monitoring - https://microlink.io https://microlink.io - automation - https://www.browserless.io https://www.browserless.io - testing - https://www.scrapingbee.com/ https://www.scrapingbee.com/ - scraping All are business leveraging these types of frameworks
- mrskitch 6y agoThanks for the mention (browserless.io)! Really enjoying this site, let me know if you'd like a guest post and I'd happily add something. We've gathered a few weird tricks and tips as well. Definitely a lot that can be shared!
- deleted 6y ago[deleted]
- jmnicolas 6y agoI don't like to have my home WiFi on all the time but I want to be able to activate / desactivate it from my phone anytime so I send a SMS to my RaspberryPi (running raspisms) that launches an app that connects to my ISP router box and activate the WiFi like I would do manually. My app is powered by Selenium but the concept is the same.
- bharani_m 6y agoI use Puppeteer at https://www.emailthis.me https://www.emailthis.me for scraping meaningful content from web pages and converting web pages to PDF.
- gadders 6y agoScalping tickets: https://www.theregister.com/2019/03/01/ticket_scalper_bots/ https://www.theregister.com/2019/03/01/ticket_scalper_bots/
- tnolet 6y agoMaybe fun to add. The guides and articles are all on GitHub as is the source code for the site - it’s Vuepress based so it might help folks who want to make their own knowledge base using that framework.
- fareesh 6y agoHow does one reliably use puppeteer and know when a page has loaded? I've tried using the various networkidle events, wait for some DOM element, and I find myself just using 5 seconds or something like that as the most reliable solution. Is there a foolproof way of doing this? I feel like it should be way easier and less hacky.
- tudorconstantin 6y agoI don't think that's possible in the most generic way. Think about when would you say the Facebook page (post login) is fully loaded. You've got there the infinite scroll, the chat box, the notifications. All of those are never fully loaded.
- chromedev 6y agohttps://pptr.dev/#?product=Puppeteer&version=v5.2.1&show=api-pagegotourl-options https://pptr.dev/#?product=Puppeteer&version=v5.2.1&show=api... You can also look for a specific element.
- luckylion 6y agoYou can't really know whether a site has fully loaded and is just sending analytics ajax to the server or is querying some random service and will trigger a navigation based on the response, whether there's a setTimeout running somewhere that'll trigger a navigation etc. I've found a timeout to be the safest bet. If onload has been fired and it hasn't jumped in the last X seconds, it's probably going to stay at that URL. Humans don't know either, they're just better at guessing based on visual cues.
- ethbro 6y agoIt's something of a UX question too. One could design a page that visually loaded, then jumped to a redirect after 10 seconds on the page. But who would? The primary approach is always event-based, because most pages do that sanely. If not... the best approach I've found is looking for sentinel elements. Essentially, something that only matches once the website is de facto loaded (regardless of events). Sometimes it's a "search results found" bit of text, sometimes a first element. But more or less, "How do I (a human) know when the page is ready?"
- ffpip 6y agoYou might wanna switch to another analytics provider (temporarily) when posting on HN. Because 60%+ users probably block it anyway, so you will lose the insights, etc
- janOsch 6y agoI am asking out of curiosity: is it possible & legal to set up a simple proxy server to redirect analytics data to the provider? So that the analytics traffic goes to the same domain - and is not blocked?
- spanhandler 6y agoGoogle probably don't want that happening because they use the data to inform decisions for all kinds of other things (search, ad sales, filling in gaps in their creepy profiles of people, maybe even metrics-driven machine-learning-for-web-design, who knows) and are more worried about web site owners feeding them mountains of fake data than about missing stats on some % of ad-blocking users.
- ffpip 6y agoUse uBlock Origin. It blocks even those sneaky DNS changers
- ffpip 6y agouBlock Origin blocks that too. CNAME blocking. But only on Firefox, since Chrome doesnt support it. When on example.com , a request is sent to definitely-not-tracking.example.com . But it just going to tracking.ad-server.com . uBO blocks those on Firefox https://medium.com/nextdns/cname-cloaking-the-dangerous-disguise-of-third-party-trackers-195205dc522a https://medium.com/nextdns/cname-cloaking-the-dangerous-disg... https://github.com/uBlockOrigin/uBlock-issues/issues/780 https://github.com/uBlockOrigin/uBlock-issues/issues/780
- tudorconstantin 6y agoWhat I'd like an article about is how to have e2e tests that also collect test coverage for apps written in react with node on the backend. Even more so if the stack is using typescript. And an added bonus for client, server and the tests running each in their own container. [Shameless plug?] I've built a project starter that checks all the above points, except the testing part (at least for now). If you know how to do the testing part I'd really appreciate a PR: https://github.com/tudorconstantin/knests https://github.com/tudorconstantin/knests EDIT: added the "[Shameless plug?]" part.
- chromedev 6y agoThis is about headless scripting for the browser, but your comment appears to be a shameless self-promotion for something unrelated.
- tudorconstantin 6y agoI thought it's about a knowledge base about e2e testing using playwright/puppeteer. The problem I wrote about is one that's solvable using those tools (I think and hope). It's a problem I'm trying to solve in my limited free time and I was hoping someone already have solved it, but if that's not the case and someone also finds it interesting, how should've I let people know there already is an open source playground for this, without posting the project.
- chromedev 6y agoI get that but when you brought up TypeScript and React, these tools are pretty indifferent to the underlying frameworks and are primarily used to simulate web browser interactions. I suppose you could integrate them into your e2e testing automatically, but every site is different so it would likely be difficult to do anything useful.
- tudorconstantin 6y agoI mentioned typescript and react (meaning jsx actually) because browsers run javascript. So the browser will have to be able to make a mapping from javascript -> jsx -> typescript in order to report correctly what was covered during the execution.
- chromedev 6y agoI've used and love Puppeteer, but it also makes me realize with enough money and the right skills, these tools are exactly what would be used to mass manipulate social media. You could literally create thousands or millions of accounts and add enough entropy to make it undetectable. I've created a couple scripts to delete accounts, and even signing in can include randomization between scrolls and clicks to make each change entirely unique and to mimic real user interactions. Sort of scary to think about what is possible with this, especially given a large pool of residential IP addresses.
- bryanrasmussen 6y ago>with enough money and the right skills, these tools are exactly what would be used to mass manipulate social media. I believe with enough money and the right skills it has always been possible to manipulate the media - social or otherwise. The question is if puppeteer means we can decrease the amount of money significantly for a minor increase in the amount of skills.
- chromedev 6y agoAbsolutely, there are already cheap third-party captcha solvers that basically outsource the captchas to real people. I could also see companies using this to inflate their user count.
- tnolet 6y agoOP here: funny and true story. We had a misconfiguration in our rate limiting. One smarty pants used it to blast his twitch channel with "viewers", e.g. a 1000 concurrent Puppeteer sessions. It was probably great for his engagement / viewership numbers till we shut him down.
- sovietmudkipz 6y agoI have to ask... What was the reason you shut this person down? Was it simply that they were violating your rate limit, or something more? E.g. if they were paying for the 1000 concurrent puppeteer sessions, would everything be in the clear with your SaaS? Presumably, your service doesn't care what the users use it for. Sure, though, it's a violation of twitch's terms of service to fake viewers on the platform. I may be naive -- could twitch sue a service that is used to fake viewers?
- umaar 6y agoI'm working on a video course for all things browser automation, whether it's testing, scraping, auditing, deploying etc. I'm keeping the codebase open on GitHub: https://github.com/umaar/learn-browser-testing/ https://github.com/umaar/learn-browser-testing/ so anyone who wants to follow along can do so for free. I've almost finished some cool content such as: - An Amazon price checker which sends you a text message when the price decreases - An Playwright script which gets to Wikipedia Philosophy (https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosophy https://en.wikipedia.org/wiki/Wikipedia:Getting_to_Philosoph...) - Having an automation script constantly running on a cheap Raspberry PI
- moooo99 6y agoA good use case would be indexing real estate portals for interested buyers. I could imagine it as a relatively attractive side business, targeting real estate agencies and buyers/renters alike. But I haven't had the time to actually build it
- mxuribe 6y agoFrom my having worked ~8 years in residential real estate (and overseeing dev. and maintenance of an in-house api)...I've learned that real estate data on all those real estate sites (from zillow, to any other broker site, etc.) is very much not clean, not always correct, nor updated at a realistic interval. From a data perspective, it is extremely messy to work with. Plus real estate brokers (and related data brokers) have little incentive to clean up the data, let alone make updates more real-time. This might not be a big issue for personal apps...but just be aware of the not-so-little gaps around real estate data. Now, if someone did this as an exercise to learn about apis, scraping, web browser automation, then by all means enjoy; plenty to learn for sure! But don't expect the data to be as usuable. (Oh, and my claims of data messiness are not limited to automata, as anyone searching for homes can tell you about their disappointments when asking about a home only to hear from the agent: "Sorry, that home was sold, we/someone forgot to take the listing down...").
- randomdude402 6y ago
- lihaoyi 6y agoI used puppeteer in my book’s (https://www.handsonscala.com/ https://www.handsonscala.com/) build pipeline to convert HTML sources into PDFs, for online distribution and printing. This let me write and style my book using common web technlogies (e.g. bootstrap CSS) rather than needing to fiddle with specialized tooling like Pandoc or LaTeX, and it ended up looking pretty good. Works flawlessly, and has exactly the configuration knobs you would expect and want. Took a bit of plumbing to call into a Node.js script from my Scala build logic, but all in all ended up being like 20 lines of plumbing which was straightforward to write and understand. A welcome change after struggling with bugs in wkhtmltopdf!
- rossta 6y agoThis is a really interesting use case! Screenshots of your book look great. I'd just started looking into the existing tools like those you mentioned for e-book authoring and it's quickly gotten overwhelming. From what you've described, I'm interested in looking into puppeteer as an alternative workflow.
- gitgud 6y agoAnybody know of a framework which can record user sessions and output puppeteer commands? (basically mouse clicks -> code) It would make writing end-2-end integration tests much easier...
- hlenke 6y agoHey, here is a basic & open source recorder for Puppeteer: https://github.com/checkly/puppeteer-recorder https://github.com/checkly/puppeteer-recorder from the same team (disclaimer I'm co-founder of Checkly)
- dpacmittal 6y agoThere's a chrome extension for nightmarejs which does exactly this
- johnmurch 6y agoTake a peak at Hackium - https://medium.com/@jsoverson/introducing-hackium-723790c00d03 https://medium.com/@jsoverson/introducing-hackium-723790c00d...
- mathias 6y agohttps://github.com/puppeteer/recorder https://github.com/puppeteer/recorder
- calderarrow 6y agoTry out the selenium IDE.[0] It's a browser extension which allows you to record your movements/actions in the browser, and then export a scaffold of the code to replicate the interactions. It could serve as a decent start point [0] https://www.selenium.dev/selenium-ide/ https://www.selenium.dev/selenium-ide/
- Dylanlacey 6y agoI work at Sauce Labs (inc. TestObject and Screener.io) and the Record-and-Playback tests are the ones that tend to be the most... Not great. They're prone to timing problems and fragile selectors. They work well for rapidly recorded, rapidly changing sites (where you can throw them away) or very stable sites (where the DOM is basically fixed), but they rapidly become a hassle in the middle. But they sure are easy. I'd suggest recording as a first pass, then going back and refactoring them to be a bit more stable; Change XPATH or structural CSS selectors to use classes and IDs, add waits and assert for page loads, etc.
- thekyle 6y agoHow do these differ from something like Selenium?
- wprapido 6y agoSelenium is multi language and multi browser and works at a bit lower level of abstraction
- ragog 6y agoThis is highly opinionated, but I have been using and teaching how to use Selenium and Appium for a few years so that is unavoidable :) I would say the pros of Puppeteer & Playwright: - generally higher speed - higher reliability (specifically lower base false-positive rate) These are mainly due to architectural choices (less moving parts between script and browser). That being said, Selenium has been the open-source standard in cross-browser testing for a long time now, and is more polished and feature-rich. Also, multi-language support makes it an easier choice for non-JS teams. I would suggest a quick hands-on POC if you want to use these tools in a project.
- eitland 6y agoSeems like this doesn't support anything but Chrome/Chromium? Edit: I was kind of wrong already and it seems I will be completely wrong soon. Which is good in this case :-) As hlenke points out below Firefox support is on its way :-)
- 6y ago
- wprapido 6y agoAwesome! Would love to see something like that for Selenium
- jadell 6y agoOne day I will right an extensive post (or set of them) about using Puppeteer to bypass sites' anti-bot measures. It's a fascinating (and annoying) cat-and-mouse game. But at the end of the day, almost all bot detection measures rely on using Javascript to report back metrics about the browser, but those measures are running in an environment where the bot completely controls what Javascript reports back. One of my favorite tricks I've seen employed are detection measures that look to see if common detection bypass tricks have been implemented (like checking the toString output of commonly overridden native functions.) https://theheadless.dev/posts/challenging-flows/#bot-detection https://theheadless.dev/posts/challenging-flows/#bot-detecti...
- chrisweekly 6y agoI know it's passe to say "hey there's an xkcd for that", but this is one of my all-time favorites, and it's directly relevant, so... enjoy! :) https://xkcd.com/810/ https://xkcd.com/810/
- ragog 6y agoPlease do! This is a very interesting topic. Looking forward to reading about it.
- rashkov 6y agoI wonder if google captcha will always be able to defeat puppeteer? Seems odd for google to publish a set of abuse-able APIs, and not be able to detect their use.
- folkhack 6y agoThere are farms of people who literally sit around all day and solve CAPTCHAs - there's no surefire way to address this problem and it usually ends up in an orchestration of reputation-score tooling (including making a user fill out a CAPTCHA) to fingerprint a bot. If you're good at spoofing all of that fingerprinting you'll blow straight past them - it's all client-side in-which you have control all the way down to the bits and bytes.
- deleted 6y ago[deleted]
- polskibus 6y agoWhat about Cypress though? How does it compare to Playwright ? Is it going to die now that MS has rolled out their own RPA?
- tnolet 6y agoCypress is an E2E testing solution with batteries included. Playwright is just the browser automation part.
- defied 6y agoI work for https://headlesstesting.com https://headlesstesting.com where we provide a grid of browsers, which people can use in combination with Puppeteer and Playwright. One of the reasons people use this, instead of Selenium, is because of the increase in speed.
- nikisweeting 6y agoI've been a long-time Puppeteer user, it's been game-changing in so many ways. We even built a whole platform with it to automate our company's social media presence, among countless small bash scripts, archiving utilities, QA workflows, etc. Playwright interests me even more though. We've been getting a lot of requests for ArchiveBox.io to support other browsers as the rendering engine for web archives, and it's always seemed daunting to try and reimplement multi-browser support ourselves for puppeteer-style workflows, but Playwright seems to completely take care of that!