13 ms·

Playwright: Automate Chromium, WebKit and Firefox

383 points by cyrusmg 5y ago

simonw 5y ago
Electron appear to have dropped support for their previous automated testing framework Spectron - https://github.com/electron-userland/spectron/blob/master/README.md https://github.com/electron-userland/spectron/blob/master/RE... - and now suggest Playwright as an alternative: https://www.electronjs.org/docs/latest/tutorial/automated-testing#using-playwright https://www.electronjs.org/docs/latest/tutorial/automated-te... and https://playwright.dev/docs/api/class-electronapplication/ https://playwright.dev/docs/api/class-electronapplication/
- eatonphil 5y ago
  WebDriver or Playwright. I switched from Spectron to selenium-webdriver.
naasking 5y ago
Anyone have experience with Playwright compared to Selenium? I have a fairly large test suite and Selenium produces constant false positive errors, typically due to various timeouts that seem fundamentally unsolvable when running it from .NET. It's just very finicky. I don't know if it's Selenium specifically or some problem with the .NET binding, but I figure Microsoft must have better .NET integration so it will at least eliminate that possible source of problems.
- mattlondon 5y ago
  I have had similar issues with selenium via other languages too - it is generally pretty flaky. E.g. saying a button or some other element doesn't exist when it clearly does. With great care and effort you can make your tests reliable (especially if you are happy to allow a "best of 3" type test strategy to allow for 1 flake and 2 passes) though. Prodigious use of the wait (i.e. stdlib polling) primitives seems to give you the most bang for your buck. I am note sure if this is just the nature of web automation, or if selenium is just crap? My gut is to say it is selenium's fault since we never get the same issues when using javascript in the DOM or in an extension)...maybe it is the browser APIs I guess? U have no idea but if this playwright is any better than that would be superb.
  - mesadb 5y ago
    I think the issue here is Selenium or Playwright they depend on Selectors which depends on UI. And when there is a change it breaks the tests. We are working on something to generate you an adapting code (Cypress first) and let you know when there needs to be a change in your test script. We have an AI model that understands the page structure as humans do. So we can do this "Click on 'Sign in' on the 'Login' page". We have a no-code tool as well which adapts to the changes. But we want to generate the code for people who want to keep things internally. Would love to discuss it more: m@preflight.com Our website: https://preflight.com https://preflight.com
- londons_explore 5y ago
  I think it's possible to write tests in selinium which are time-independant... Eg. "Wait for element #foo to exist". You can also give the browser a virtual clock so that you can use time based timeouts and give every test a timeout of 3 hours, but those 3 hours only take milliseconds in real time. That approach gets CPU expensive if your site has any background polling scripts or animation, because obviously the animation will end up animating a lot during the test!
  - naasking 5y ago
    > I think it's possible to write tests in selinium which are time-independant... Eg. "Wait for element #foo to exist". Yes, this is what I've done but the elements non-deterministically do or do not appear according to selenium, and then the wait times out. This happens for dozens of tests across dozens of pages with no issue with manual use, so something fishy is going on.
- simonw 5y ago
  On paper Playwright should be a LOT better - it's taken a similar approach to Cypress, where everything is designed around the need to reduce flaky tests. In Playwright that manifests itself as the "auto-wait" feature: https://playwright.dev/docs/actionability https://playwright.dev/docs/actionability You can do this kind of thing with Selenium too but it wasn't designed in from the very start of that project.
- i_like_apis 5y ago
  While I have not used Playwright (but have a lot of experience with Se), I would say the code style is refreshing: // Expect an element "to be visible". await expect(page.locator('text=Learn more').first()).toBeVisible(); Writing await for every action makes the timeout of the action seem more explicitly declared. There seems to be more granular control of timeouts as well https://playwright.dev/docs/test-timeouts https://playwright.dev/docs/test-timeouts > I don't know if it's Selenium specifically or some problem with the .NET binding If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it because the code execution is quicker than the page. Large Se/Webdriver suites are often a PIA. I find it's nice to write them with Python or Ruby so they can be debugged interactively with the an interactive shell.
  - naasking 5y ago
    > If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it That's what I do, but the wait for an element in certain tests times out after a few minutes, even though the elements are clearly visible, and manual use never has an issue. From other comments it sounds like Puppeteer and Playwright are better on this, so will look into switching.
    - gmokki 5y ago
      When I fixed many similar selenium/webdriver tests the root cause was always the same: You grab reference to an element and for example wait it to become enabled or some text to appear. But your ui framework actually replaces the element in the dom while doing its thing and your reference to stale element will never change. Fix is to loop searching the element with selector and check if the element fills the conditions. If not, retry from search again. We had nice helpers for those and had very stable selenium tests.
      - naasking 5y ago
        > Fix is to loop searching the element with selector and check if the element fills the conditions. If not, retry from search again. We had nice helpers for those and had very stable selenium tests. Thanks, I'll double check, but I think we do this now. In looking at the history of test failures, those failures are indeed less common, but still plenty of false positives of other types. Most persistent recent failures are the WebDriver timing out when loading a URL, which has never happened while manual testing or when being used by end users, so not sure what's going on there. In any case, if the Playwright API encourages better idioms for writing tests that avoids these pitfalls, that would be cool because I deal with a lot of work term students that aren't adept at this kind of stuff so that would save a lot of headaches.
- Bilal_io 5y ago
  I tried selenium then playwright for a .Net project, selenium wasn't easy to work with. Playwright was good but for some reason which I don't recall exactly (could have been because it had to redownload chromium everytime we deployed). I ended up switching to puppeteer and I ended up very happy with it.
  - mesadb 5y ago
    You can now generate puppeteer code from Google Chrome Recorder. You should check it out. But still it might be flaky. No-code is the best in my opinion :D https://preflight.com https://preflight.com We have done all the ground work. Like: - Concurrency - Adapt to the changes. Our selectors are like this: "Click on 'Login' button in the 'Sign in' form" - Update the tests with an HTML/Video player etc
- tmcneal 5y ago
  I'm not sure if any of these are pertinent to your tests, but these are the issues I see most often that cause flaky tests: - Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. i.e. click on a button and wait for a 'Success' message to appear. Puppeteer and Playwright both have good constructs for doing this. - Needless complexity in the tests. Conditionals in particular are a code-smell and indicate there's something needlessly complex about the test. - No test data management strategy. The more assumptions you can make about the state of your application, the simpler your tests become. Ideally tests are running in an environment that nothing else is touching, and you're seeding data into that environment before tests run. I personally don't believe in mocking data in regression tests since that quickly becomes hard to manage. We spend a lot of time thinking about these issues at my company and wrote a guide that covers other common regression testing issues in more detail here: https://reflect.run/regression-testing-guide/ https://reflect.run/regression-testing-guide/
  - naasking 5y ago
    > - Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. We don't do any timed waits, all of our waits are for an element or value to appear, but these waits never complete sometimes, non-deterministically. We then added a long 5 min timeout on these waits because we know the test will never complete at that point. It's always fine in manual testing though, and if we don't run the browser in headless mode and watch it work. Very frustrating. Sometimes the HTTP requests themselves timeout after a few minutes, but this never happens in manual testing either. That's actually the most common issue these days, and this happens non-deterministically too. This is what I meant by "flaky".
    - ethbr0 5y ago
      Does Selenium have a trace log generator that will dump out all events? I.e. all element creation on the page, matching, etc. I'm not familiar with it specifically, but that's my go-to starting place in weird automation issues like that. Normally it gives some kind of hint as to why that's happening (or why Selenium thinks it's happening).
defied 5y ago
I’m working on a project that provides remote browsers, running on VMs/containers, capable of running Playwright tests (and Puppeteer scripts): https://headlesstesting.com/ https://headlesstesting.com/ We’ve seen a consistent growth of interest in people wanting to use Playwright for browser automation (and testing).
johnnypangs 5y ago
What do people thing of playwright vs cypress? I've been considering using playwright instead as it supports more browsers and I feel like it's easier to do production monitoring (by putting it in a aws lambda or using checkly) - Cypress: https://www.cypress.io https://www.cypress.io - Playwright aws lambda: https://github.com/PauloGoncalvesBH/running-playwright-on-aws-lambda https://github.com/PauloGoncalvesBH/running-playwright-on-aw... - Checkly: https://www.checklyhq.com https://www.checklyhq.com
- dvngnt_ 5y ago
  I love Cypress. there are definitely limitations though like iframe support or visiting two separate domains, tab support https://docs.cypress.io/guides/references/trade-offs#Permanent-trade-offs-1 https://docs.cypress.io/guides/references/trade-offs#Permane...
  - deleted 5y ago
    [deleted]
- machiaweliczny 5y ago
  Playwright is definitely better IMO. Cypress is overengineered.
  - felipellrocha 5y ago
    I wouldn’t say cypress is over engineered. Just a byproduct of its time.
    - sumedh 5y ago
      What do you mean?
- CSDude 5y ago
  - Cypress does not run on M1 natively. - Playwright is more lightweight. Can be good or bad on what you expect. But I definitely prefer Playwright.
headlessvictim2 5y ago
Off-topic, but our freemium website is under attack by headless browsers. The freemium service provides access to compute-heavy machine learning models running on GPUs. Hackers blast 50-100 requests in the same second, which clog the servers and block legitimate users. We reported IPs to AWS and use Cloudflare "Super Bot Fight Mode" to thwart attacks, but the hackers still break through. We don't require accounts, but could impose account requirements if this helps. Any suggestions?
- cmeacham98 5y ago
  100 requests/second isn't that much, especially if you're fronting your website with Cloudflare. Do you have some unauthenticated endpoint(s) that eat up a ton of server CPU?
  - headlessvictim2 5y ago
    Thanks for the reply! The freemium service provides access to machine learning models on GPU instances, served with FastAPI. Each request invokes a compute-intensive ML model, but perhaps there is something wrong with the FastAPI configuration as well?
    - tempest_ 5y ago
      It could be. I watch the FastAPI repos a lot and tones of people do not understand how async python works and put their models with sync code in an async context.
      - headlessvictim2 5y ago
        Consider us one. :) We tried removing "async" -- thinking it would force sequential processing -- but it unexpectedly seemed to cause parallel processing of requests, which caused CUDA memory errors. Before removing "async", this is the weird behavior we observed: * Hacker blasts 50-100 requests. * Our ML model processes each request in normal time and sequentially. * But instead of returning individual responses immediately, the server holds onto all responses -- sending responses only when the last request finishes (or a bunch of requests finish). * Normally, request 1 should return in N seconds, request 2 in 2N seconds, but with this, all requests returned in about N50 seconds (assuming batch size of 50). 1. Any suggestions on this? 2. Mind clarifying how sync vs aync works? The FastAPI docs are unclear. Any help would be much appreciated. This has been extremely frustrating.
tw20212021 5y ago
Is there an open source web testing tool which also integrates a dashboard, keeps track of test runs, creates reports, something that I can just install on a vm and run to test a web app?
- dzhiurgis 5y ago
  playwright html reporter kinda does that, no?
- hoten 5y ago
  Give Lighthouse CI a shot. https://github.com/GoogleChrome/lighthouse-ci https://github.com/GoogleChrome/lighthouse-ci
robstain 5y ago
Anyone tried BotCity? https://botcity.dev https://botcity.dev
- petermd 5y ago
vthommeret 5y ago
Reposting my previous notes on Playwright (https://news.ycombinator.com/item?id=30060135 https://news.ycombinator.com/item?id=30060135): I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev https://playwright.dev It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it: 1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the DOM, visible, stable (not animating), can receive events, and are enabled: https://playwright.dev/docs/actionability https://playwright.dev/docs/actionability 2. It's fast — It creates multiple processes and runs tests in parallel, unlike e.g. Cypress. 3. It's cross-browser — supports Chrome, Safari, and Firefox out-of-the-box. 4. The tracing tools are incredible, you can step through the entire test execution and get a live DOM that you can inspect with your browser's existing developer tools, see all console.logs, etc... 5. The developers and community are incredibly responsive. This is one of the biggest ones — issues are quickly responded to and addressed often by the founders, pull requests are welcomed and Slack is highly active and respectful. My prior experience with end-to-end tests was that they were highly buggy and unreliable and so Playwright was a welcome surprise and inspired me to fully test all the variations of our checkout flow.
- deleted 5y ago
  [deleted]
syspec 5y ago
Interesting tidbit: One of the main contributors of this project[0], was the core contributor (creator?) of Puppeteer[1], but then I guess left Google to join Microsoft and work on this[2][3]. [0] - https://github.com/aslushnikov https://github.com/aslushnikov [1] - https://github.com/puppeteer/puppeteer/ https://github.com/puppeteer/puppeteer/ [2] - https://github.com/microsoft/playwright/graphs/contributors https://github.com/microsoft/playwright/graphs/contributors [3] - https://github.com/microsoft/playwright/graphs/contributors https://github.com/microsoft/playwright/graphs/contributors
- vlunkr 5y ago
  I wonder what the story is there. Why wouldn't MS just have him continue to work on Puppeteer? They're both open source, so there's not much point in "owning" their own clone of it.
  - fikama 5y ago
    From what I know puppeter works only with chromium, this could be deal breaker for microsoft
    - jahewson 5y ago
      Edge is chromium though.
      - zamadatix 5y ago
        Microsoft's web products run on more than Edge.
        wizzwizz4 5y ago
        They try very hard to stop you trying, though; Microsoft Teams has removed the “continue without the Teams app” button entirely on Firefox, even though I'm fairly sure it still works fine in Firefox.
forgotmyoldacc 5y ago
Benefits of Playwright over Puppeteer - official support for languages outside of JavaScript, and official codegen/record support. Great!
- machiaweliczny 5y ago
  Also testing on safari/iphone is easy. It also has built-in snapshot testing. I just wish it was integrated with S3 as git LFS is not good
ithrow 5y ago
For generating PDFs like invoices in a webapp, is libraries like this the way to go these days or is still using a pdf lib the norm? Pros of Playwright/Puppeteer: Reuse existing HTML/CSS knowledge Cons: Requires an external service or shelling out to an external process Pros of using a pdf lib: Probably better performance, simpler architecture by being in-process. Cons: Ad-hoc language for designing the PDF.
- deleted 5y ago
  [deleted]
- rudasn 5y ago
  It's very simple to use either, there are loads of example implementations on GitHub. I used one based on docker, and the bottleneck was actually sending the html, css you want to print (if it's not already served over http). I used a shared docker volume to write to from one process (python) and read from another (the node pupetter). It all comes down to, load html, wait to load, save to pdf. Very simple, fast, and reliable. More so than weasyprint for example.
kundi 5y ago
Does it support screencast - video recording of the browser with audio?
- mxschmitt 5y ago
  It supports video recording (without audio), screenshots, and post mortem recording which is called Tracing.
  - dzhiurgis 5y ago
    I recognise your name from playwright! Thanks for the product, I love it. All of the above are uploaded automatically as Github artefact as part of auto-generated Github Action!
brimstedt 5y ago
Does anyone know how it compares to NightwatchJs? Br
Karupan 5y ago
Playwright is great, especially if you are dealing with test cases that span multiple domains/contexts. I had to test some user flows which involved logging into two apps, each with three different users to perform and validate various actions. Playwright's context switching made it a breeze. Also, it offers a nice separation of browser automation and test runner API, so it can be used outside of E2E testing as well.
hoten 5y ago
Having worked with these folks back in Chrome, it's been great seeing this project continue to be successful. Great job!
apatheticonion 5y ago
The only thing I wish we had was remote browser access - so I could run my tests on a VM (like within a docker image) and use a browser on the host. We use TestCafe at work for this purpose. I personally hate TestCafe as it's is an absurd unfocused mess of a browser remote, but it lets me control my browser by navigating to a URL which no other browser remote system does.
- kesor 5y ago
  You can do that with X11, set the display of your desktop inside the docker container and playwright's browser will appear on your desktop.
thenerdhead 5y ago
Playwright is a great tool. I was able to create a proof-of-concept stock screening tool using automation & screenshots of HTML elements to help me get swing trading ideas each morning/night when the market closed. It's a .NET Core console app using a CLI library called spectre.console based on rich(python) and playwright as the workhorse. There's so much potential to use playwright in CI/CD with GitHub Actions cron jobs. Really enjoying it so far.
tzs 5y ago
How hard is it for a site (server side or via JavaScript on the page) to tell that it is being accessed via a browser that is being automated with Playwright? I've seen some sites that behave differently when the browser is being automated. E.g., if I access fanfiction.net from a browser being automated with Selenium it gets stuck in an endless Cloudflare CAPTCHA loop. Accordingly I've come to prefer automation methods that are less revealing to the site.
slimeboty 5y ago
shp0ngle 5y ago
Looking at the source, I wonder why lots of the files have Google copyright? https://github.com/microsoft/playwright/blob/0d277fa589e9508c04de38a4e4806d595f2134bf/packages/playwright-core/src/server/firefox/ffPage.ts https://github.com/microsoft/playwright/blob/0d277fa589e9508... edit: ah puppeteer was a Google project. I forgot
jwithington 5y ago
Are there any products for QA folks that reduce the workload? I find most things are still done manually…
- tmcneal 5y ago
  Yes definitely, there's lots of products in the QA space trying to tackle the problem you're describing. I'm a co-founder of a no-code product in the space (https://reflect.run https://reflect.run). Being no-code has the advantage of enabling all QA testers to build test automation, regardless of coding experience.
- mesadb 5y ago
  We are also in the space. Manual testing is definitely time consuming. You can automate your manual testing with https://preflight.com https://preflight.com Would love to help any of your testing needs
tomcam 5y ago
What is the … operator for? test.use({ ...devices['iPhone 13 Pro'], locale: 'en-US', geolocation: { longitude: 12.492507, latitude: 41.889938 }, permissions: ['geolocation'], })
- tycoon177 5y ago
  That is the JavaScript spread operator. It takes all of the elements of an object or an array and adds them to another. In this case, the code posted is merging the iphone 13 pro device settings object into an object literal
  - tomcam 5y ago
    Most cool, thx
deleted 5y ago
[deleted]
cfrover 5y ago
There is also this Github Discussions thread on how it compares with things like Cypress, which is a E2E testing tool used by lots of people on the frontend web. TLDR: Playwright can achieve nearly all the things cypress can & more due to it being a fully scriptable browser - https://github.com/microsoft/playwright/discussions/11201 https://github.com/microsoft/playwright/discussions/11201 - https://cathalmacdonnacha.com/cypress-vs-playwright-which-is-best-for-e2e-testing https://cathalmacdonnacha.com/cypress-vs-playwright-which-is... - https://alisterbscott.com/2021/10/27/five-reasons-why-playwright-is-better-than-cypress/ https://alisterbscott.com/2021/10/27/five-reasons-why-playwr...

383 points by cyrusmg 5y ago

Electron appear to have dropped support for their previous automated testing framework Spectron - https://github.com/electron-userland/spectron/blob/master/README.md https://github.com/electron-userland/spectron/blob/master/RE... - and now suggest Playwright as an alternative: https://www.electronjs.org/docs/latest/tutorial/automated-testing#using-playwright https://www.electronjs.org/docs/latest/tutorial/automated-te... and https://playwright.dev/docs/api/class-electronapplication/ https://playwright.dev/docs/api/class-electronapplication/

eatonphil 5y ago
WebDriver or Playwright. I switched from Spectron to selenium-webdriver.

naasking 5y ago

Anyone have experience with Playwright compared to Selenium? I have a fairly large test suite and Selenium produces constant false positive errors, typically due to various timeouts that seem fundamentally unsolvable when running it from .NET. It's just very finicky. I don't know if it's Selenium specifically or some problem with the .NET binding, but I figure Microsoft must have better .NET integration so it will at least eliminate that possible source of problems.

mattlondon 5y ago
I have had similar issues with selenium via other languages too - it is generally pretty flaky. E.g. saying a button or some other element doesn't exist when it clearly does. With great care and effort you can make your tests reliable (especially if you are happy to allow a "best of 3" type test strategy to allow for 1 flake and 2 passes) though. Prodigious use of the wait (i.e. stdlib polling) primitives seems to give you the most bang for your buck. I am note sure if this is just the nature of web automation, or if selenium is just crap? My gut is to say it is selenium's fault since we never get the same issues when using javascript in the DOM or in an extension)...maybe it is the browser APIs I guess? U have no idea but if this playwright is any better than that would be superb.
- mesadb 5y ago
  I think the issue here is Selenium or Playwright they depend on Selectors which depends on UI. And when there is a change it breaks the tests. We are working on something to generate you an adapting code (Cypress first) and let you know when there needs to be a change in your test script. We have an AI model that understands the page structure as humans do. So we can do this "Click on 'Sign in' on the 'Login' page". We have a no-code tool as well which adapts to the changes. But we want to generate the code for people who want to keep things internally. Would love to discuss it more: m@preflight.com Our website: https://preflight.com https://preflight.com
londons_explore 5y ago
I think it's possible to write tests in selinium which are time-independant... Eg. "Wait for element #foo to exist". You can also give the browser a virtual clock so that you can use time based timeouts and give every test a timeout of 3 hours, but those 3 hours only take milliseconds in real time. That approach gets CPU expensive if your site has any background polling scripts or animation, because obviously the animation will end up animating a lot during the test!
- naasking 5y ago
  > I think it's possible to write tests in selinium which are time-independant... Eg. "Wait for element #foo to exist". Yes, this is what I've done but the elements non-deterministically do or do not appear according to selenium, and then the wait times out. This happens for dozens of tests across dozens of pages with no issue with manual use, so something fishy is going on.
simonw 5y ago
On paper Playwright should be a LOT better - it's taken a similar approach to Cypress, where everything is designed around the need to reduce flaky tests. In Playwright that manifests itself as the "auto-wait" feature: https://playwright.dev/docs/actionability https://playwright.dev/docs/actionability You can do this kind of thing with Selenium too but it wasn't designed in from the very start of that project.
i_like_apis 5y ago
While I have not used Playwright (but have a lot of experience with Se), I would say the code style is refreshing: // Expect an element "to be visible". await expect(page.locator('text=Learn more').first()).toBeVisible(); Writing await for every action makes the timeout of the action seem more explicitly declared. There seems to be more granular control of timeouts as well https://playwright.dev/docs/test-timeouts https://playwright.dev/docs/test-timeouts > I don't know if it's Selenium specifically or some problem with the .NET binding If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it because the code execution is quicker than the page. Large Se/Webdriver suites are often a PIA. I find it's nice to write them with Python or Ruby so they can be debugged interactively with the an interactive shell.
- naasking 5y ago
  > If the execution in .NET is slow then I suppose it could be .NET. But it could be (and often is) the suite design. You must wait for /everything/ before interacting with it That's what I do, but the wait for an element in certain tests times out after a few minutes, even though the elements are clearly visible, and manual use never has an issue. From other comments it sounds like Puppeteer and Playwright are better on this, so will look into switching.
  - gmokki 5y ago
    When I fixed many similar selenium/webdriver tests the root cause was always the same: You grab reference to an element and for example wait it to become enabled or some text to appear. But your ui framework actually replaces the element in the dom while doing its thing and your reference to stale element will never change. Fix is to loop searching the element with selector and check if the element fills the conditions. If not, retry from search again. We had nice helpers for those and had very stable selenium tests.
    - naasking 5y ago
      > Fix is to loop searching the element with selector and check if the element fills the conditions. If not, retry from search again. We had nice helpers for those and had very stable selenium tests. Thanks, I'll double check, but I think we do this now. In looking at the history of test failures, those failures are indeed less common, but still plenty of false positives of other types. Most persistent recent failures are the WebDriver timing out when loading a URL, which has never happened while manual testing or when being used by end users, so not sure what's going on there. In any case, if the Playwright API encourages better idioms for writing tests that avoids these pitfalls, that would be cool because I deal with a lot of work term students that aren't adept at this kind of stuff so that would save a lot of headaches.
Bilal_io 5y ago
I tried selenium then playwright for a .Net project, selenium wasn't easy to work with. Playwright was good but for some reason which I don't recall exactly (could have been because it had to redownload chromium everytime we deployed). I ended up switching to puppeteer and I ended up very happy with it.
- mesadb 5y ago
  You can now generate puppeteer code from Google Chrome Recorder. You should check it out. But still it might be flaky. No-code is the best in my opinion :D https://preflight.com https://preflight.com We have done all the ground work. Like: - Concurrency - Adapt to the changes. Our selectors are like this: "Click on 'Login' button in the 'Sign in' form" - Update the tests with an HTML/Video player etc
tmcneal 5y ago
I'm not sure if any of these are pertinent to your tests, but these are the issues I see most often that cause flaky tests: - Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. i.e. click on a button and wait for a 'Success' message to appear. Puppeteer and Playwright both have good constructs for doing this. - Needless complexity in the tests. Conditionals in particular are a code-smell and indicate there's something needlessly complex about the test. - No test data management strategy. The more assumptions you can make about the state of your application, the simpler your tests become. Ideally tests are running in an environment that nothing else is touching, and you're seeding data into that environment before tests run. I personally don't believe in mocking data in regression tests since that quickly becomes hard to manage. We spend a lot of time thinking about these issues at my company and wrote a guide that covers other common regression testing issues in more detail here: https://reflect.run/regression-testing-guide/ https://reflect.run/regression-testing-guide/
- naasking 5y ago
  > - Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. We don't do any timed waits, all of our waits are for an element or value to appear, but these waits never complete sometimes, non-deterministically. We then added a long 5 min timeout on these waits because we know the test will never complete at that point. It's always fine in manual testing though, and if we don't run the browser in headless mode and watch it work. Very frustrating. Sometimes the HTTP requests themselves timeout after a few minutes, but this never happens in manual testing either. That's actually the most common issue these days, and this happens non-deterministically too. This is what I meant by "flaky".
  - ethbr0 5y ago
    Does Selenium have a trace log generator that will dump out all events? I.e. all element creation on the page, matching, etc. I'm not familiar with it specifically, but that's my go-to starting place in weird automation issues like that. Normally it gives some kind of hint as to why that's happening (or why Selenium thinks it's happening).

defied 5y ago

I’m working on a project that provides remote browsers, running on VMs/containers, capable of running Playwright tests (and Puppeteer scripts): https://headlesstesting.com/ https://headlesstesting.com/ We’ve seen a consistent growth of interest in people wanting to use Playwright for browser automation (and testing).

johnnypangs 5y ago

What do people thing of playwright vs cypress? I've been considering using playwright instead as it supports more browsers and I feel like it's easier to do production monitoring (by putting it in a aws lambda or using checkly) - Cypress: https://www.cypress.io https://www.cypress.io - Playwright aws lambda: https://github.com/PauloGoncalvesBH/running-playwright-on-aws-lambda https://github.com/PauloGoncalvesBH/running-playwright-on-aw... - Checkly: https://www.checklyhq.com https://www.checklyhq.com

dvngnt_ 5y ago
I love Cypress. there are definitely limitations though like iframe support or visiting two separate domains, tab support https://docs.cypress.io/guides/references/trade-offs#Permanent-trade-offs-1 https://docs.cypress.io/guides/references/trade-offs#Permane...
- deleted 5y ago
  [deleted]
machiaweliczny 5y ago
Playwright is definitely better IMO. Cypress is overengineered.
- felipellrocha 5y ago
  I wouldn’t say cypress is over engineered. Just a byproduct of its time.
  - sumedh 5y ago
    What do you mean?
CSDude 5y ago
- Cypress does not run on M1 natively. - Playwright is more lightweight. Can be good or bad on what you expect. But I definitely prefer Playwright.

headlessvictim2 5y ago

Off-topic, but our freemium website is under attack by headless browsers. The freemium service provides access to compute-heavy machine learning models running on GPUs. Hackers blast 50-100 requests in the same second, which clog the servers and block legitimate users. We reported IPs to AWS and use Cloudflare "Super Bot Fight Mode" to thwart attacks, but the hackers still break through. We don't require accounts, but could impose account requirements if this helps. Any suggestions?

cmeacham98 5y ago
100 requests/second isn't that much, especially if you're fronting your website with Cloudflare. Do you have some unauthenticated endpoint(s) that eat up a ton of server CPU?
- headlessvictim2 5y ago
  Thanks for the reply! The freemium service provides access to machine learning models on GPU instances, served with FastAPI. Each request invokes a compute-intensive ML model, but perhaps there is something wrong with the FastAPI configuration as well?
  - tempest_ 5y ago
    It could be. I watch the FastAPI repos a lot and tones of people do not understand how async python works and put their models with sync code in an async context.
    - headlessvictim2 5y ago
      Consider us one. :) We tried removing "async" -- thinking it would force sequential processing -- but it unexpectedly seemed to cause parallel processing of requests, which caused CUDA memory errors. Before removing "async", this is the weird behavior we observed: * Hacker blasts 50-100 requests. * Our ML model processes each request in normal time and sequentially. * But instead of returning individual responses immediately, the server holds onto all responses -- sending responses only when the last request finishes (or a bunch of requests finish). * Normally, request 1 should return in N seconds, request 2 in 2N seconds, but with this, all requests returned in about N50 seconds (assuming batch size of 50). 1. Any suggestions on this? 2. Mind clarifying how sync vs aync works? The FastAPI docs are unclear. Any help would be much appreciated. This has been extremely frustrating.

tw20212021 5y ago

Is there an open source web testing tool which also integrates a dashboard, keeps track of test runs, creates reports, something that I can just install on a vm and run to test a web app?

dzhiurgis 5y ago
playwright html reporter kinda does that, no?
hoten 5y ago
Give Lighthouse CI a shot. https://github.com/GoogleChrome/lighthouse-ci https://github.com/GoogleChrome/lighthouse-ci

robstain 5y ago

Anyone tried BotCity? https://botcity.dev https://botcity.dev

petermd 5y ago

vthommeret 5y ago

Reposting my previous notes on Playwright (https://news.ycombinator.com/item?id=30060135 https://news.ycombinator.com/item?id=30060135): I just want to plug Playwright by Microsoft as I've been using it over the past month and have had a really great experience with it: https://playwright.dev https://playwright.dev It's built by the founders of Puppeteer which came out of the Chrome team. Some things I like about it: 1. It's reliable and implements auto-waiting as described in the article. You can use modern async/await syntax and it ensures elements are a) attached to the DOM, visible, stable (not animating), can receive events, and are enabled: https://playwright.dev/docs/actionability https://playwright.dev/docs/actionability 2. It's fast — It creates multiple processes and runs tests in parallel, unlike e.g. Cypress. 3. It's cross-browser — supports Chrome, Safari, and Firefox out-of-the-box. 4. The tracing tools are incredible, you can step through the entire test execution and get a live DOM that you can inspect with your browser's existing developer tools, see all console.logs, etc... 5. The developers and community are incredibly responsive. This is one of the biggest ones — issues are quickly responded to and addressed often by the founders, pull requests are welcomed and Slack is highly active and respectful. My prior experience with end-to-end tests was that they were highly buggy and unreliable and so Playwright was a welcome surprise and inspired me to fully test all the variations of our checkout flow.

deleted 5y ago
[deleted]

syspec 5y ago

Interesting tidbit: One of the main contributors of this project[0], was the core contributor (creator?) of Puppeteer[1], but then I guess left Google to join Microsoft and work on this[2][3]. [0] - https://github.com/aslushnikov https://github.com/aslushnikov [1] - https://github.com/puppeteer/puppeteer/ https://github.com/puppeteer/puppeteer/ [2] - https://github.com/microsoft/playwright/graphs/contributors https://github.com/microsoft/playwright/graphs/contributors [3] - https://github.com/microsoft/playwright/graphs/contributors https://github.com/microsoft/playwright/graphs/contributors

vlunkr 5y ago
I wonder what the story is there. Why wouldn't MS just have him continue to work on Puppeteer? They're both open source, so there's not much point in "owning" their own clone of it.
- fikama 5y ago
  From what I know puppeter works only with chromium, this could be deal breaker for microsoft
  - jahewson 5y ago
    Edge is chromium though.
    - zamadatix 5y ago
      Microsoft's web products run on more than Edge.
      - wizzwizz4 5y ago
        They try very hard to stop you trying, though; Microsoft Teams has removed the “continue without the Teams app” button entirely on Firefox, even though I'm fairly sure it still works fine in Firefox.

forgotmyoldacc 5y ago

Benefits of Playwright over Puppeteer - official support for languages outside of JavaScript, and official codegen/record support. Great!

machiaweliczny 5y ago
Also testing on safari/iphone is easy. It also has built-in snapshot testing. I just wish it was integrated with S3 as git LFS is not good

ithrow 5y ago

For generating PDFs like invoices in a webapp, is libraries like this the way to go these days or is still using a pdf lib the norm? Pros of Playwright/Puppeteer: Reuse existing HTML/CSS knowledge Cons: Requires an external service or shelling out to an external process Pros of using a pdf lib: Probably better performance, simpler architecture by being in-process. Cons: Ad-hoc language for designing the PDF.

deleted 5y ago
[deleted]
rudasn 5y ago
It's very simple to use either, there are loads of example implementations on GitHub. I used one based on docker, and the bottleneck was actually sending the html, css you want to print (if it's not already served over http). I used a shared docker volume to write to from one process (python) and read from another (the node pupetter). It all comes down to, load html, wait to load, save to pdf. Very simple, fast, and reliable. More so than weasyprint for example.

kundi 5y ago

Does it support screencast - video recording of the browser with audio?

mxschmitt 5y ago
It supports video recording (without audio), screenshots, and post mortem recording which is called Tracing.
- dzhiurgis 5y ago
  I recognise your name from playwright! Thanks for the product, I love it. All of the above are uploaded automatically as Github artefact as part of auto-generated Github Action!

brimstedt 5y ago

Does anyone know how it compares to NightwatchJs? Br

Karupan 5y ago

Playwright is great, especially if you are dealing with test cases that span multiple domains/contexts. I had to test some user flows which involved logging into two apps, each with three different users to perform and validate various actions. Playwright's context switching made it a breeze. Also, it offers a nice separation of browser automation and test runner API, so it can be used outside of E2E testing as well.

hoten 5y ago

Having worked with these folks back in Chrome, it's been great seeing this project continue to be successful. Great job!

apatheticonion 5y ago

The only thing I wish we had was remote browser access - so I could run my tests on a VM (like within a docker image) and use a browser on the host. We use TestCafe at work for this purpose. I personally hate TestCafe as it's is an absurd unfocused mess of a browser remote, but it lets me control my browser by navigating to a URL which no other browser remote system does.

kesor 5y ago
You can do that with X11, set the display of your desktop inside the docker container and playwright's browser will appear on your desktop.

thenerdhead 5y ago

Playwright is a great tool. I was able to create a proof-of-concept stock screening tool using automation & screenshots of HTML elements to help me get swing trading ideas each morning/night when the market closed. It's a .NET Core console app using a CLI library called spectre.console based on rich(python) and playwright as the workhorse. There's so much potential to use playwright in CI/CD with GitHub Actions cron jobs. Really enjoying it so far.

tzs 5y ago

How hard is it for a site (server side or via JavaScript on the page) to tell that it is being accessed via a browser that is being automated with Playwright? I've seen some sites that behave differently when the browser is being automated. E.g., if I access fanfiction.net from a browser being automated with Selenium it gets stuck in an endless Cloudflare CAPTCHA loop. Accordingly I've come to prefer automation methods that are less revealing to the site.

slimeboty 5y ago

shp0ngle 5y ago

Looking at the source, I wonder why lots of the files have Google copyright? https://github.com/microsoft/playwright/blob/0d277fa589e9508c04de38a4e4806d595f2134bf/packages/playwright-core/src/server/firefox/ffPage.ts https://github.com/microsoft/playwright/blob/0d277fa589e9508... edit: ah puppeteer was a Google project. I forgot

jwithington 5y ago

Are there any products for QA folks that reduce the workload? I find most things are still done manually…

tmcneal 5y ago
Yes definitely, there's lots of products in the QA space trying to tackle the problem you're describing. I'm a co-founder of a no-code product in the space (https://reflect.run https://reflect.run). Being no-code has the advantage of enabling all QA testers to build test automation, regardless of coding experience.
mesadb 5y ago
We are also in the space. Manual testing is definitely time consuming. You can automate your manual testing with https://preflight.com https://preflight.com Would love to help any of your testing needs

tomcam 5y ago

What is the … operator for? test.use({ ...devices['iPhone 13 Pro'], locale: 'en-US', geolocation: { longitude: 12.492507, latitude: 41.889938 }, permissions: ['geolocation'], })

tycoon177 5y ago
That is the JavaScript spread operator. It takes all of the elements of an object or an array and adds them to another. In this case, the code posted is merging the iphone 13 pro device settings object into an object literal
- tomcam 5y ago
  Most cool, thx

deleted 5y ago

[deleted]

cfrover 5y ago

There is also this Github Discussions thread on how it compares with things like Cypress, which is a E2E testing tool used by lots of people on the frontend web. TLDR: Playwright can achieve nearly all the things cypress can & more due to it being a fully scriptable browser - https://github.com/microsoft/playwright/discussions/11201 https://github.com/microsoft/playwright/discussions/11201 - https://cathalmacdonnacha.com/cypress-vs-playwright-which-is-best-for-e2e-testing https://cathalmacdonnacha.com/cypress-vs-playwright-which-is... - https://alisterbscott.com/2021/10/27/five-reasons-why-playwright-is-better-than-cypress/ https://alisterbscott.com/2021/10/27/five-reasons-why-playwr...