12 ms·

Web automation: Don't use Selenium, use Playwright

408 points by rekahrv 4y ago

starik36 4y ago
Unfortunately, the .NET library doesn't support ARM processors. Thus, I can't use it on the Raspberry Pi. But PuppeteerSharp works great on the Pi.
ffgh 4y ago
I find that Hero[https://ulixee.org/docs/hero https://ulixee.org/docs/hero] has been a good replacement for my selenium needs
- porjo 4y ago
  Thanks. I've found Chromedp [0] to be a good Selenium replacement when programming in Go. Used in conjunction with headless-shell [1] you can deploy a Go app into a container and do the testing all within the same container with low overhead. [0] https://github.com/chromedp/chromedp https://github.com/chromedp/chromedp [1] https://github.com/chromedp/docker-headless-shell https://github.com/chromedp/docker-headless-shell
999900000999 4y ago
Playwright is good, but stop thinking auto generated code is worthwhile. Eventually you'll have to write your own scripts. I will admit Playwright can spit out code that will at least serve as a starting point.
- cebert 4y ago
  If you use automation attributes on your components it can work quite well.
geoelectric 4y ago
You're going to get a lot of responses that say record-replay testing doesn't scale, is unmaintainable, etc. They're totally right. But the thing is, early on when your app is in flux, neither does writing Selenium code. There's a pretty big truism in UI automation that writing UI tests before a UI freeze is a recipe for a shit-ton of pain. Coding to ids or xpaths only gets you yay far if the UI flow itself fundamentally changes. But re-recording might be easy. Don't use stuff like this for long-standing tests. Unless you architect your app just right so that IDs are always stable, and you never change the interface, it'll break and break in ways that you can only fix by a complete re-record. Plus the tests tend to be extremely timing-fragile because recording uses delays instead of synchronization points, so they just won't work in CI. But do use stuff like this at your desk during bring-up, when the cost of a re-record is lower than the cost of a test rewrite and it's ok to re-run the thing if the first try craps out due to a timing glitch. And from there, keep an open mind. I went to a GTAC (Google testing conference) where a presentation made a very good argument--with numbers and everything--that for smaller projects with simple and more or less static UIs, and where the tests were all fundamentally scripted, there was almost no advantage to coding the tests later. Record-replay was the best way to go. But I definitely don't think a system like PlayWright fully replaces remote stubs like WebDriver and coding tests in the languages that can talk to it. At some point you hit the issue that the login screen changed and now every single test is invalid. It's awfully nice if you used Page Object Model and you have a single-point-of-truth to fix it. More to the point, test automation can do more than just execute a script repeatably. Randomized monkey testing is something you can really only do in code, ditto scenarios that need to talk to testing interfaces exposed from the app itself. Glad you found a tool that resonates with you!
- pydry 4y ago
  Record-replay doesnt even work when running the same scenario 2 minutes later half the time - never mind in 2 months' time. I tend to find in most apps that unless you directly change the front end templates to give elements you interact with better identifiers and then deliberately use those identifiers your browser tests will always end up an unreliable sucky mess.
  - geoelectric 4y ago
    Agreed, giving it at least stable identifiers to play with makes a huge difference. With some front end generators, that may not be trivial, though I'd think Selenium being a thing (POM does rely heavily on stable unique identifiers, since "nth element past m" xpath queries are way more fragile) has probably made it more of a standard ask. Dynamically-created UIs are usually the hardest thing to deal with--if they get a different ID every time, record-replay is out the door, and even Selenium is a lot harder. But personally, with a lot of automation architecture experience, I think you're exaggerating a little re: doesn't work 2 minutes later half the time. But it also depends on whether your app is driving some invisible external resource that makes the timing highly variable, etc. Even then you can usually "harden" the recording by putting in worst-case delays. It's just that then your test takes so long to run you can't CI it either. It's really situational. What I'm arguing against more than anything is the knee-jerk reaction that it's never appropriate. It's just never enough usually. The fact that we tend to skip over the option entirely in testing is probably a blind spot and a mistake. Devs dabbling in testing as a side task definitely shouldn't ignore the option.
- MetaWhirledPeas 4y ago
  > It's awfully nice if you used Page Object Model and you have a single-point-of-truth to fix it. I like fewer sources of truth, but I dislike trying to force everything into a "page" analogy. Try maintaining a library of commonly-used actions instead, like: logIn() registerNewUser() addItemToCart() And maybe group them thematically. The underlying intent is the same... reduce duplication, improve organization. But you won't be stuck wondering "what page am I on now?" in a single page application, or rifling through folder upon folder of nonsensical "page" files. And even in the worst case, Ctrl-Shift-H still works.
lucasfcosta 4y ago
I’ve used playwright for a while, especially on my day job. It’s a fantastic and versatile tool. I couldn’t agree more.
kc10 4y ago
> I found weird issues (when using the recorder) where I couldn't scroll down to the bottom of the screen in Playwright's Chromium, but could in a normal Chrome, forcing me to use other tricks to click the button. You may consider trying this extension - https://chrome.google.com/webstore/detail/devraven-recorder/fonaldkpejilcpapnkcfecppoognbcah https://chrome.google.com/webstore/detail/devraven-recorder/.... This extension can record the tests using your Chrome browser instead of launching a separate Playwright chromium browser. Full disclosure: I am the developer of this extension.
cdrini 4y ago
I'm always surprised to see posts like this, and no one mentioning testcafe. It's a super solid e2e testing library! I've been using it for quite a while and it runs reliably and consistently. The API design can be a little messy, but it was (and still is!) a top contender when I was evaluating tools in this space. And open source and contributor friendly. Not sure why I so rarely see it mentioned!
- Sysctl232 4y ago
  Indeed, I evaluated the e2e landscape years ago and TestCafe was the clear winner. It's a solid solution that integrates well with my TypeScript front and backend.
andrewmcwatters 4y ago
For those of you who are as confused as I am about Playwright's growth over Selenium: https://star-history.com/#microsoft/playwright&SeleniumHQ/selenium&Date https://star-history.com/#microsoft/playwright&SeleniumHQ/se... My company uses Selenium for some of our projects, but I evaluated the browser automation space some time ago and found that it was the best-in-class solution at that time. It looks like things have since changed.
- jsdalton 4y ago
  Same chart, including Cypress: https://star-history.com/#microsoft/playwright&SeleniumHQ/selenium&cypress-io/cypress&Date https://star-history.com/#microsoft/playwright&SeleniumHQ/se...
jbaczuk 4y ago
Yeah, but python? eww :)
- number6 4y ago
  Also has nodejs binding
- brynjolf 4y ago
  It also has C# bindings, even though I think the C# documentation is a bit lacking compared to node.js and python
azangru 4y ago
What happened to Puppeteer? I barely see it mentioned anywhere now.
- mountaineer 4y ago
  > The Puppeteer team essentially moved from Google to Microsoft and became the Playwright team. https://blog.logrocket.com/playwright-vs-puppeteer/ https://blog.logrocket.com/playwright-vs-puppeteer/
- andrewmcwatters 4y ago
  Never knew why people bothered with Puppeteer, you need to test against other browsers, too. Not just Chrome.
throwaway0asd 4y ago
Selenium, Playwright, and Puppeteer all make use of Chrome DevTools Protocol (CDP) to access the browser, even Firefox. CDP is the current de facto standard to remotely access the browser. It actually starts an HTTP server in the browser that you send requests to. The W3C is working on actual standard for this space that will likely be more mature and durable. My learning from playing CDP and the test automation applications mentioned is that this is way harder (and slower) than test automation should be and it’s a huge pain in the ass for authoring tests. The experience is so bad that it only seems to appeal to professional testers whose jobs depend upon it. To solve for that I wrote my own test automation solution that does not require remote access to the browser. In my personal application where I use this form of testing I am able to execute about 30 user events per second in the browser. That performance speed is completely dependent upon other performance conditions of your application, hardware, and transmission handling. The test cases are simple data structures defined as TypeScript interfaces in a Node application communicating to the browser over WebSockets. The events are executed by creating a new Event object in the browser and executed with the event’s dispatchEvent property. When your automation gets really fast (less than 30 seconds for a major test suite) everybody will make use of it including the developers, product owners, and even the business leaders. It becomes a faster way to setup and access various features of a browser application than doing it manually.
- deleted 4y ago
  [deleted]
- quickthrower2 4y ago
  > To solve for that I wrote my own test automation solution that does not require remote access to the browser. Any more details? Are you still working with actual chrome?
  - throwaway0asd 4y ago
    I cannot provide too many details without violating anonymity of this anonymous account. I can say that in my primary test suite I have about 85 tests each which comprise a variable number of user events and assertions totaling about 290 total assertions. When I was using HTTP that would take about 45 seconds to execute. When I switched to a home grown websockets solution the test time went down to about 6.5-8.5 seconds. Also, if you want things to be lightning fast try to shy away from using querySelectors.
peterweyand 4y ago
Weird question. The trouble with web automation tools such as Selenium and beautiful soup is that they require specific instructions to parse html. So it's difficult to auto scrape websites if several websites have a different html layout or any website simply decides to change it's html. Would it be possible to create a neural network model that could be trained to parse html as a human would? Or an AI for that matter that could break Google's captcha. I'm surprised that that still works given the advances in image recognition. If both are possible, the security on the web may quickly become a thing of the past.
- melony 4y ago
  https://www.adept.ai/act https://www.adept.ai/act You can combine traditional RPA with modern ML and do very interesting things.
mountaintimefrm 4y ago
Playwright is the bee knees. At my last tech job I did maintenance on about 400 wordpress sites, and I used playwright to automate a bunch of monotonous clicky tasks normally done via the wordpress backend gui. It was a lot of fun and toward the end I got playwright running on heroku and made a gui so that my not very tech coworkers could upload a csv of different sites and then and a select a dropdown list of different scrapers, tests, etc.
mmmpetrichor 4y ago
As a long time test automation dev, I would never want to be tasked with e2e browser automation again. I did that for a brief period in my career. It didn't feel like real coding. The tests always seemed like they were barely less maintenance than manual testing. I'm curious how big the e2e web tests suites are for best in breed web software. My opinion has been to minimize the amount of e2e browser tests and do app integration tests solely against the API (assuming there is one!), but I've never specialized in web app testing.
- eastbound 4y ago
  Web frameworks are big enough that even if the API is successful, the UI may be wrong. It’s extremely useful to have good enough UI tests that the dev himself can see what is failing. See that another way: You, as a human, can’t scale for a team of 50. e2e tests can.
  - deleted 4y ago
    [deleted]
  - MetaWhirledPeas 4y ago
    > Web frameworks are big enough that even if the API is successful, the UI may be wrong. It’s extremely useful to have good enough UI tests that the dev himself can see what is failing. This is absolutely true. Who cares if the JSON payload is correct if the page is wrong?
- MetaWhirledPeas 4y ago
  > I would never want to be tasked with e2e browser automation again This is how I felt before I started using Cypress. You need a tool that helps you rather than being a drag on you, and you need to commit some time to it. It's helped me learn a lot of coding I didn't otherwise know. Writing the test itself is usually not "real coding" but writing helper methods and learning how to organize your code certainly is. The trick to making it all work is making your tests not flaky. They need to pass 99% of the time. Now we have a few hundred long end-to-end tests and maybe 2 of them are flaky at any given time. They are easy to fix when they break, and they uncover important bugs on a regular basis. Granted, I'm in the business of testing. Anything that isn't automated I have to do manually. And I have to represent a real user, so simple API tests are not going to cut it, and I have an incentive to automate the entire end-to-end process with multiple variations. But if you want to do short API tests you can certainly do that, and Cypress is actually a nice tool for that too.
recroad 4y ago
Beyond perhaps a smoke/availability test, I don’t know why you’d perform UI testing. Too brittle, too slow, too much of a pain to maintain compared to the value.
- returnInfinity 4y ago
  how do you perform automated in your build pipeline? I am genuinely curious and looking for new ways to do things.
- Sysctl232 4y ago
  I like UI tests as the test are so thorough. You test the entire stack from database to webserver and final page render. Say you test a sign-up form. The next test logs into a mail account to check if the welcome email has arrived, checking the job scheduler, 3th party email service and if the link in the email works as intended. I don't mind a bit of brittleness given the large test surface e2e tests cover.
- epolanski 4y ago
  You do so you can go sleep knowing you are making money and nothing breaks. There are no tests as valuable as E2E tests.
mherrmann 4y ago
OP struggles with XPaths in complex DOMs. This is a solved problem. Use Helium [1] to write test code that acts on what the user sees: start_chrome('github.com/login') write('user', into='Username') write('pw', into='Password') click('Sign in') I am the author of Helium. It's a wrapper around Selenium. It's fully open source. Under the hood, it uses Selenium 3, which is old. But boy does it work beautifully. 1: https://github.com/mherrmann/selenium-python-helium https://github.com/mherrmann/selenium-python-helium
- maneesh 4y ago
  Wow this is cool
- ZeKK14 4y ago
  Does it solves the issues with virtual dom ? It's a pain to manage. The readme mentions iframes but not that.
- irjustin 4y ago
  Super cool, but to be fair to the author, Clicking does seem easier still.
  - mherrmann 4y ago
    Until the web site changes and you have to re-record.
    - CharlieDigital 4y ago
      Playwright's selectors are very resilient. Yes, your copy might change, but that is also a change in spec (whereas a change in DOM is not always a change in spec).
    - eastendguy 4y ago
      Yes! Recording xpath works fine in many cases, but once something on the website changes you are left with this auto-generated Xpath that you do not know how to change. Maintaining such a test is a nightmare and a waste of brain-cycles.
meitham 4y ago
OP suggests that having a large company behind a software tool (Microsoft in full control of Playright) is a positive thing and is an advantage over Selenium. I tend to see that negatively. Community driven projects (or non profit organisation behind them) like Python, Postgres, Firefox always are more trustworthy and likely to last longer in general.
- matai_kolila 4y ago
  You won't like learning where Guido's been working for many years now...
  - meitham 4y ago
    I find it astonishing that you thinking the current employer of the BDFL isn’t public knowledge! But even more astonishing that you alluding this can have an impact on the PSF. GvR working for Microsoft isn’t the same as Microsoft acquistion of GitHub.
    - pjmlp 4y ago
      Well, CPython performance only started to matter after Guido got hired by Microsoft, the wonders of big corps.
      - solarkraft 4y ago
        Having big corporations contribute to projects is totally great! The incentives are just fucky when they control them. My favorite example: VSCode is a great piece of software, but almost half of it is actually only available in a proprietary build from Microsoft's website that contains telemetry.
        thiht 4y ago
        > but almost half of it is actually only available in a proprietary build from Microsoft Which half? VSCodium works great, so I'd go as far to say 99% of VSCode is actually open source (the name and icons being exceptions). You cannot say proprietary extensions are half of VSCode.
jve 4y ago
> Selenium just couldn't parse the over-engineered Javascript framework we had, and I had to pile on hacks on hacks. Excuse me, but isn't selenium actually a browser robot and not a javascript parser? I assume this is just that DOM is mutating so much that they could or could not find that element or always have to wait for it and comes with unreliability and such. I haven't used alternatives, but using selenium feels like solid "Engine" to make abstractions on.
- CalRobert 4y ago
  That's odd - Selenium has been my go-to when I need to scrape and Scrapy won't suffice (I'm not doing anything too huge in scale) and it handles web apps well. I do have a lot of waiting for elements to be visible, admittedly.
  - Finnucane 4y ago
    Testing, scraping, and automating are not really the same tasks.
    - epolanski 4y ago
      I'm a heavy user of playwright and I find it by far the best tool for all of three.
EugeneOZ 4y ago
I’m here just to say how much I love Playwright :) Best features for me: - Auto-waiting - it is a BRILLIANT feature! - Shared authentication; - Supported browsers list; - API testing; - Tests generator (with recognition of data-test-id attributes); - Flaky tests detection; - Flexible config for recording failed tests; - Ease of installation and updates; - Great performance. Thank you, Playwright team!
ur-whale 4y ago
None of these solutions are robust wrt highly dynamic web sites or - worse - in-depth changes to the HTML implemented over time server-side. What all of these tools actually need is to be bundled with some AI-powered recognition tools - computer vision, semantic understanding, etc ... - that can sort of "understand" the web page served before interacting with it. All people do right now is a combo of XPATH-based search for an element in the DOM, or if that fails some sort of regex based pattern matching on the HTML. These are weak and brittle techniques that are pretty much guaranteed to fail at some point.
- sentrms 4y ago
  I've changed from WordPress to Vue to React but can still use my login e2e test as I depend on a DOM element ID. Eg: #username_field, #password_field, #submit_btn.
felixhummel 4y ago
Using TLS client certificates automatically is still not possible in Playwright [1]. That's our main reason why we keep using Selenium (with pytest-selenium) and Selenium Grid [2]. [1] https://github.com/microsoft/playwright/issues/1799 https://github.com/microsoft/playwright/issues/1799 [2] https://www.selenium.dev/documentation/grid/ https://www.selenium.dev/documentation/grid/
a_c 4y ago
Actually what is stopping you from generating automation selenium script by recording browser action? I recently wrote a tiny browser extension to do just that. Intercept event and translate the action into a selenium script. Currently the supported features are few. But would love to learn what am I expecting down the path.
jawr 4y ago
What’s the best way of monitoring suites? The built in report tool is amazing, but would love a solution that makes for easier analysis and maybe a closer integration with jira.
tmd83 4y ago
Can someone provide an in depth comparison between Cypress and Playwright? It seems that both of them could be good choice compared to Selenium. For example I keep seeing auto-wait being talked about for Playwright but doesn't cypress have that too? It seems playwright has better browser support and multi-window/concurrency story and cypress have better built-in tools (like a test-runner UI). Basically looking for recommendation for test-automation for an old complex app mostly written using server-side tech but has potential for react and such moving forward.
- MetaWhirledPeas 4y ago
  I haven't used Playwright, but I can tell just by reading about it that the code has more boilerplate and doesn't read as nicely. Example of a Cypress test (pardon the freehand typing here): describe(('the google home page') => { it(('allows you to search for banana pictures') => { cy.visit('https://google.com') cy.get('[aria-label="Search"]').type('banana{enter}') cy.contains('Images').click() cy.get('[alt="Image result for banana"]').should('be.visible') }) }) Nothing too weird, and a lot of the library functions you'd need are available globally without any importing. It even has jQuery and Lodash.
- epolanski 4y ago
  Playwright supports webkit by default, even in your linux CI. Playwright is faster. It's fully free and OS. It's written by Microsoft to test their own thousands of services not to sell you anything like Cypress. The authors of Playwright are the leads from Puppeteeer, Lighthouse, the first node debugger and Chrome Dev Tools, the engineering team has simply no comparisons in the browser automation field.
lyu07282 4y ago
My journey has been Selenium, then Cypress, then Playwright. This was the first time I was happy writing ui tests, Playwright is great! (Though I don't use the recording tool at all this article focuses on I rather write the tests manually)
- baal80spam 4y ago
  > Though I don't use the recording tool at all this article focuses on I rather write the tests manually That's because automatic code generators rarely work (anything more dynamically generated and it's a no-go).
  - GenerocUsername 4y ago
    I manually do it, then replace parameters where needed
- dvngnt_ 4y ago
  i miss the cypress test runner
  - lyu07282 4y ago
    The thing about cypress that annoyed me the most was how it wasn't just using async/await syntax, generating more complex tests dynamically is super hard. Just try to write a test that follows links on a site dynamically, I couldn't figure it out.
    - btown 4y ago
      Cypress requires developers to have a mental model of "calling a Cypress function actually adds an action to the back of a queue and immediately returns control flow without waiting for that action to run." And this is fine if you can express your waits and conditions declaratively - you give Cypress a plan, it decodes the plan from your Cypress code in milliseconds, and then will take as long as it needs to execute that plan. And Cypress gives a great API surface where almost everything can be done with just this declarative API, and it enables them to show the full plan in a UI even while it's executing. But if you actually need custom control flow, with code you control... you need to "pause" your action-enqueueing to let Cypress catch up. Which means you're going to be nesting callbacks, or sprinkling awaits in all the places that aren't logically awaiting, because you need an explicit yield point out of your JS block. Our team finds this rare enough that it's acceptable. But... yeah, it's far from a perfect solution.
plugin-baby 4y ago
I’ve tried writing e2e tests for web apps using browser drivers on-and-off for the last decade, and it feels like with playwright that the tech has finally matured enough to make the tests reliable enough to be worth the effort of maintaining them.
- sh4rks 4y ago
  What does Playwright do that Selenium can't do?
  - ncrmro 4y ago
    Run inside a container with near zero effort for CI pipelines
  - voxic11 4y ago
    The main difference is that Playwright uses custom headless builds of "browser projects" (like chromium) rather than attempting to automate a normal consumer build of the browser via the webdriver api. The webdriver api is kind of slow, browsers don't always implement it well, and it doesn't have a good mechanism for the browser to push events back to the test application which ends up requiring selenium to do a lot of polling for changes so it can miss events that happen quickly.
    - ethbr0 4y ago
      Events are the bane of automation existence. Back in Windows-heavyweight app land, it made way more sense to interpose a low overhead shim to intercept and copy events. I can't imagine the pain you'd encounter trying to paper over the other way around.
      - imiric 4y ago
        > Events are the bane of automation existence. They do make some tasks tricky to handle, like tracking state, or avoiding race conditions, but in reality events are a good fit for interacting with browser DOM events, representing browser activity, and being able to react to it. Consider the complexity of an average modern web site, that loads dozens of scripts, uses a frontend framework, is highly dynamic and interactive (likely a SPA), contains multiple frames, etc. All of this happens concurrently, and the only abstraction that makes sense to represent it is an event-based system. You can't achieve the same level of responsiveness using a traditional request/response protocol. This is why modern browser automation protocols are event-based, and built on top of WebSocket. Selenium is pushing for WebDriver BiDi, while the browser standard is the Chrome DevTools Protocol.
MetaWhirledPeas 4y ago
> So much so, I started looking at Javascript testing frameworks like Chai, Mocha, Cypress etc. The problem I found is that they require a completely different setup, and aren't easy to get started for someone from a Python background. A shame to see Cypress lumped in and dismissed like that. It really is a fantastic way to test. > The killer feature of Playwright is: You can automatically generate tests by opening a web browser and manually running through the steps you want. It saves the hassle I faced with Selenium, where you were opening developer tools and finding the Xpath or similar. Yuck This is absolutely not the primary hassle with testing. Recording steps can help kick start your testing, sure. But very quickly you start to realize that it's only saving you from the easiest work at best, and is creating flaky tests at worst.
- salawat 4y ago
  Indeed. You need: loggers, data generators, integrations with state probe points, data/result aggregators, etc, etc. For massive numbers of tests, now it's time to look into parallelism. Really hope you're not sharing your test data between test cases! Or holding open a single db session to your write instance! How about multimedia artifacts management? Point being, test automators do a shit ton of work nobody ever wants to dig into. Tests are not there to only be written. They must be read.
- sdfhbdf 4y ago
  > Cypress lumped in and dismissed like that I think Cypress is really cool when you're on their happy path - testing on Chromium browsers and in JavaScript. I tried it some time ago so maybe this might be outdated, but they had little support for browsers other than Chromium or even rather Electron based and also it was a little difficult to wire up with Gherkin scenarios. Keep in mind that the OP refers to Python tools, AFAIK Cypress does not offer Python bindings while both Selenium and Playwright do.
  - jjzhiyuan 4y ago
    We have used Cypress for 3 years to test our oss project: https://apisix.apache.org/blog/2021/02/08/stable-product-delivery-with-cypress/ https://apisix.apache.org/blog/2021/02/08/stable-product-del...
irsagent 4y ago
It must be a game changer as Selenium is mentioned, but never linked in the article.
simonw 4y ago
I just spent half an hour wrestling with Selenium for Python and that API has NOT aged well. content = driver.find_element(By.CSS_SELECTOR, "p.content") In Playwright Python: content = page.locator("p.content") Playwright is just nicer to use. Selenium feels very Java-ish.
- KyeRussell 4y ago
  Yeah, not only that, but that’s the NEWER Selenium Python API. The older style was actually (imo) easier to grapple with, discover, etc. The whole thing has felt like it’s either written by people that don’t work with Python very often, or that it’s too married to the idea of being a close port of an API targeted at…yeah…Java.

408 points by rekahrv 4y ago

Unfortunately, the .NET library doesn't support ARM processors. Thus, I can't use it on the Raspberry Pi. But PuppeteerSharp works great on the Pi.

ffgh 4y ago

I find that Hero[https://ulixee.org/docs/hero https://ulixee.org/docs/hero] has been a good replacement for my selenium needs

porjo 4y ago
Thanks. I've found Chromedp [0] to be a good Selenium replacement when programming in Go. Used in conjunction with headless-shell [1] you can deploy a Go app into a container and do the testing all within the same container with low overhead. [0] https://github.com/chromedp/chromedp https://github.com/chromedp/chromedp [1] https://github.com/chromedp/docker-headless-shell https://github.com/chromedp/docker-headless-shell

999900000999 4y ago

Playwright is good, but stop thinking auto generated code is worthwhile. Eventually you'll have to write your own scripts. I will admit Playwright can spit out code that will at least serve as a starting point.

cebert 4y ago
If you use automation attributes on your components it can work quite well.

geoelectric 4y ago

You're going to get a lot of responses that say record-replay testing doesn't scale, is unmaintainable, etc. They're totally right. But the thing is, early on when your app is in flux, neither does writing Selenium code. There's a pretty big truism in UI automation that writing UI tests before a UI freeze is a recipe for a shit-ton of pain. Coding to ids or xpaths only gets you yay far if the UI flow itself fundamentally changes. But re-recording might be easy. Don't use stuff like this for long-standing tests. Unless you architect your app just right so that IDs are always stable, and you never change the interface, it'll break and break in ways that you can only fix by a complete re-record. Plus the tests tend to be extremely timing-fragile because recording uses delays instead of synchronization points, so they just won't work in CI. But do use stuff like this at your desk during bring-up, when the cost of a re-record is lower than the cost of a test rewrite and it's ok to re-run the thing if the first try craps out due to a timing glitch. And from there, keep an open mind. I went to a GTAC (Google testing conference) where a presentation made a very good argument--with numbers and everything--that for smaller projects with simple and more or less static UIs, and where the tests were all fundamentally scripted, there was almost no advantage to coding the tests later. Record-replay was the best way to go. But I definitely don't think a system like PlayWright fully replaces remote stubs like WebDriver and coding tests in the languages that can talk to it. At some point you hit the issue that the login screen changed and now every single test is invalid. It's awfully nice if you used Page Object Model and you have a single-point-of-truth to fix it. More to the point, test automation can do more than just execute a script repeatably. Randomized monkey testing is something you can really only do in code, ditto scenarios that need to talk to testing interfaces exposed from the app itself. Glad you found a tool that resonates with you!

pydry 4y ago
Record-replay doesnt even work when running the same scenario 2 minutes later half the time - never mind in 2 months' time. I tend to find in most apps that unless you directly change the front end templates to give elements you interact with better identifiers and then deliberately use those identifiers your browser tests will always end up an unreliable sucky mess.
- geoelectric 4y ago
  Agreed, giving it at least stable identifiers to play with makes a huge difference. With some front end generators, that may not be trivial, though I'd think Selenium being a thing (POM does rely heavily on stable unique identifiers, since "nth element past m" xpath queries are way more fragile) has probably made it more of a standard ask. Dynamically-created UIs are usually the hardest thing to deal with--if they get a different ID every time, record-replay is out the door, and even Selenium is a lot harder. But personally, with a lot of automation architecture experience, I think you're exaggerating a little re: doesn't work 2 minutes later half the time. But it also depends on whether your app is driving some invisible external resource that makes the timing highly variable, etc. Even then you can usually "harden" the recording by putting in worst-case delays. It's just that then your test takes so long to run you can't CI it either. It's really situational. What I'm arguing against more than anything is the knee-jerk reaction that it's never appropriate. It's just never enough usually. The fact that we tend to skip over the option entirely in testing is probably a blind spot and a mistake. Devs dabbling in testing as a side task definitely shouldn't ignore the option.
MetaWhirledPeas 4y ago
> It's awfully nice if you used Page Object Model and you have a single-point-of-truth to fix it. I like fewer sources of truth, but I dislike trying to force everything into a "page" analogy. Try maintaining a library of commonly-used actions instead, like: logIn() registerNewUser() addItemToCart() And maybe group them thematically. The underlying intent is the same... reduce duplication, improve organization. But you won't be stuck wondering "what page am I on now?" in a single page application, or rifling through folder upon folder of nonsensical "page" files. And even in the worst case, Ctrl-Shift-H still works.

lucasfcosta 4y ago

I’ve used playwright for a while, especially on my day job. It’s a fantastic and versatile tool. I couldn’t agree more.

kc10 4y ago

> I found weird issues (when using the recorder) where I couldn't scroll down to the bottom of the screen in Playwright's Chromium, but could in a normal Chrome, forcing me to use other tricks to click the button. You may consider trying this extension - https://chrome.google.com/webstore/detail/devraven-recorder/fonaldkpejilcpapnkcfecppoognbcah https://chrome.google.com/webstore/detail/devraven-recorder/.... This extension can record the tests using your Chrome browser instead of launching a separate Playwright chromium browser. Full disclosure: I am the developer of this extension.

cdrini 4y ago

I'm always surprised to see posts like this, and no one mentioning testcafe. It's a super solid e2e testing library! I've been using it for quite a while and it runs reliably and consistently. The API design can be a little messy, but it was (and still is!) a top contender when I was evaluating tools in this space. And open source and contributor friendly. Not sure why I so rarely see it mentioned!

Sysctl232 4y ago
Indeed, I evaluated the e2e landscape years ago and TestCafe was the clear winner. It's a solid solution that integrates well with my TypeScript front and backend.

andrewmcwatters 4y ago

For those of you who are as confused as I am about Playwright's growth over Selenium: https://star-history.com/#microsoft/playwright&SeleniumHQ/selenium&Date https://star-history.com/#microsoft/playwright&SeleniumHQ/se... My company uses Selenium for some of our projects, but I evaluated the browser automation space some time ago and found that it was the best-in-class solution at that time. It looks like things have since changed.

jsdalton 4y ago
Same chart, including Cypress: https://star-history.com/#microsoft/playwright&SeleniumHQ/selenium&cypress-io/cypress&Date https://star-history.com/#microsoft/playwright&SeleniumHQ/se...

jbaczuk 4y ago

Yeah, but python? eww :)

number6 4y ago
Also has nodejs binding
brynjolf 4y ago
It also has C# bindings, even though I think the C# documentation is a bit lacking compared to node.js and python

azangru 4y ago

What happened to Puppeteer? I barely see it mentioned anywhere now.

mountaineer 4y ago
> The Puppeteer team essentially moved from Google to Microsoft and became the Playwright team. https://blog.logrocket.com/playwright-vs-puppeteer/ https://blog.logrocket.com/playwright-vs-puppeteer/
andrewmcwatters 4y ago
Never knew why people bothered with Puppeteer, you need to test against other browsers, too. Not just Chrome.

throwaway0asd 4y ago

Selenium, Playwright, and Puppeteer all make use of Chrome DevTools Protocol (CDP) to access the browser, even Firefox. CDP is the current de facto standard to remotely access the browser. It actually starts an HTTP server in the browser that you send requests to. The W3C is working on actual standard for this space that will likely be more mature and durable. My learning from playing CDP and the test automation applications mentioned is that this is way harder (and slower) than test automation should be and it’s a huge pain in the ass for authoring tests. The experience is so bad that it only seems to appeal to professional testers whose jobs depend upon it. To solve for that I wrote my own test automation solution that does not require remote access to the browser. In my personal application where I use this form of testing I am able to execute about 30 user events per second in the browser. That performance speed is completely dependent upon other performance conditions of your application, hardware, and transmission handling. The test cases are simple data structures defined as TypeScript interfaces in a Node application communicating to the browser over WebSockets. The events are executed by creating a new Event object in the browser and executed with the event’s dispatchEvent property. When your automation gets really fast (less than 30 seconds for a major test suite) everybody will make use of it including the developers, product owners, and even the business leaders. It becomes a faster way to setup and access various features of a browser application than doing it manually.

deleted 4y ago
[deleted]
quickthrower2 4y ago
> To solve for that I wrote my own test automation solution that does not require remote access to the browser. Any more details? Are you still working with actual chrome?
- throwaway0asd 4y ago
  I cannot provide too many details without violating anonymity of this anonymous account. I can say that in my primary test suite I have about 85 tests each which comprise a variable number of user events and assertions totaling about 290 total assertions. When I was using HTTP that would take about 45 seconds to execute. When I switched to a home grown websockets solution the test time went down to about 6.5-8.5 seconds. Also, if you want things to be lightning fast try to shy away from using querySelectors.

peterweyand 4y ago

Weird question. The trouble with web automation tools such as Selenium and beautiful soup is that they require specific instructions to parse html. So it's difficult to auto scrape websites if several websites have a different html layout or any website simply decides to change it's html. Would it be possible to create a neural network model that could be trained to parse html as a human would? Or an AI for that matter that could break Google's captcha. I'm surprised that that still works given the advances in image recognition. If both are possible, the security on the web may quickly become a thing of the past.

melony 4y ago
https://www.adept.ai/act https://www.adept.ai/act You can combine traditional RPA with modern ML and do very interesting things.

mountaintimefrm 4y ago

Playwright is the bee knees. At my last tech job I did maintenance on about 400 wordpress sites, and I used playwright to automate a bunch of monotonous clicky tasks normally done via the wordpress backend gui. It was a lot of fun and toward the end I got playwright running on heroku and made a gui so that my not very tech coworkers could upload a csv of different sites and then and a select a dropdown list of different scrapers, tests, etc.

mmmpetrichor 4y ago

As a long time test automation dev, I would never want to be tasked with e2e browser automation again. I did that for a brief period in my career. It didn't feel like real coding. The tests always seemed like they were barely less maintenance than manual testing. I'm curious how big the e2e web tests suites are for best in breed web software. My opinion has been to minimize the amount of e2e browser tests and do app integration tests solely against the API (assuming there is one!), but I've never specialized in web app testing.

eastbound 4y ago
Web frameworks are big enough that even if the API is successful, the UI may be wrong. It’s extremely useful to have good enough UI tests that the dev himself can see what is failing. See that another way: You, as a human, can’t scale for a team of 50. e2e tests can.
- deleted 4y ago
  [deleted]
- MetaWhirledPeas 4y ago
  > Web frameworks are big enough that even if the API is successful, the UI may be wrong. It’s extremely useful to have good enough UI tests that the dev himself can see what is failing. This is absolutely true. Who cares if the JSON payload is correct if the page is wrong?
MetaWhirledPeas 4y ago
> I would never want to be tasked with e2e browser automation again This is how I felt before I started using Cypress. You need a tool that helps you rather than being a drag on you, and you need to commit some time to it. It's helped me learn a lot of coding I didn't otherwise know. Writing the test itself is usually not "real coding" but writing helper methods and learning how to organize your code certainly is. The trick to making it all work is making your tests not flaky. They need to pass 99% of the time. Now we have a few hundred long end-to-end tests and maybe 2 of them are flaky at any given time. They are easy to fix when they break, and they uncover important bugs on a regular basis. Granted, I'm in the business of testing. Anything that isn't automated I have to do manually. And I have to represent a real user, so simple API tests are not going to cut it, and I have an incentive to automate the entire end-to-end process with multiple variations. But if you want to do short API tests you can certainly do that, and Cypress is actually a nice tool for that too.

recroad 4y ago

Beyond perhaps a smoke/availability test, I don’t know why you’d perform UI testing. Too brittle, too slow, too much of a pain to maintain compared to the value.

returnInfinity 4y ago
how do you perform automated in your build pipeline? I am genuinely curious and looking for new ways to do things.
Sysctl232 4y ago
I like UI tests as the test are so thorough. You test the entire stack from database to webserver and final page render. Say you test a sign-up form. The next test logs into a mail account to check if the welcome email has arrived, checking the job scheduler, 3th party email service and if the link in the email works as intended. I don't mind a bit of brittleness given the large test surface e2e tests cover.
epolanski 4y ago
You do so you can go sleep knowing you are making money and nothing breaks. There are no tests as valuable as E2E tests.

mherrmann 4y ago

OP struggles with XPaths in complex DOMs. This is a solved problem. Use Helium [1] to write test code that acts on what the user sees: start_chrome('github.com/login') write('user', into='Username') write('pw', into='Password') click('Sign in') I am the author of Helium. It's a wrapper around Selenium. It's fully open source. Under the hood, it uses Selenium 3, which is old. But boy does it work beautifully. 1: https://github.com/mherrmann/selenium-python-helium https://github.com/mherrmann/selenium-python-helium

maneesh 4y ago
Wow this is cool
ZeKK14 4y ago
Does it solves the issues with virtual dom ? It's a pain to manage. The readme mentions iframes but not that.
irjustin 4y ago
Super cool, but to be fair to the author, Clicking does seem easier still.
- mherrmann 4y ago
  Until the web site changes and you have to re-record.
  - CharlieDigital 4y ago
    Playwright's selectors are very resilient. Yes, your copy might change, but that is also a change in spec (whereas a change in DOM is not always a change in spec).
  - eastendguy 4y ago
    Yes! Recording xpath works fine in many cases, but once something on the website changes you are left with this auto-generated Xpath that you do not know how to change. Maintaining such a test is a nightmare and a waste of brain-cycles.

meitham 4y ago

OP suggests that having a large company behind a software tool (Microsoft in full control of Playright) is a positive thing and is an advantage over Selenium. I tend to see that negatively. Community driven projects (or non profit organisation behind them) like Python, Postgres, Firefox always are more trustworthy and likely to last longer in general.

matai_kolila 4y ago
You won't like learning where Guido's been working for many years now...
- meitham 4y ago
  I find it astonishing that you thinking the current employer of the BDFL isn’t public knowledge! But even more astonishing that you alluding this can have an impact on the PSF. GvR working for Microsoft isn’t the same as Microsoft acquistion of GitHub.
  - pjmlp 4y ago
    Well, CPython performance only started to matter after Guido got hired by Microsoft, the wonders of big corps.
    - solarkraft 4y ago
      Having big corporations contribute to projects is totally great! The incentives are just fucky when they control them. My favorite example: VSCode is a great piece of software, but almost half of it is actually only available in a proprietary build from Microsoft's website that contains telemetry.
      - thiht 4y ago
        > but almost half of it is actually only available in a proprietary build from Microsoft Which half? VSCodium works great, so I'd go as far to say 99% of VSCode is actually open source (the name and icons being exceptions). You cannot say proprietary extensions are half of VSCode.

jve 4y ago

> Selenium just couldn't parse the over-engineered Javascript framework we had, and I had to pile on hacks on hacks. Excuse me, but isn't selenium actually a browser robot and not a javascript parser? I assume this is just that DOM is mutating so much that they could or could not find that element or always have to wait for it and comes with unreliability and such. I haven't used alternatives, but using selenium feels like solid "Engine" to make abstractions on.

CalRobert 4y ago
That's odd - Selenium has been my go-to when I need to scrape and Scrapy won't suffice (I'm not doing anything too huge in scale) and it handles web apps well. I do have a lot of waiting for elements to be visible, admittedly.
- Finnucane 4y ago
  Testing, scraping, and automating are not really the same tasks.
  - epolanski 4y ago
    I'm a heavy user of playwright and I find it by far the best tool for all of three.

EugeneOZ 4y ago

I’m here just to say how much I love Playwright :) Best features for me: - Auto-waiting - it is a BRILLIANT feature! - Shared authentication; - Supported browsers list; - API testing; - Tests generator (with recognition of data-test-id attributes); - Flaky tests detection; - Flexible config for recording failed tests; - Ease of installation and updates; - Great performance. Thank you, Playwright team!

ur-whale 4y ago

None of these solutions are robust wrt highly dynamic web sites or - worse - in-depth changes to the HTML implemented over time server-side. What all of these tools actually need is to be bundled with some AI-powered recognition tools - computer vision, semantic understanding, etc ... - that can sort of "understand" the web page served before interacting with it. All people do right now is a combo of XPATH-based search for an element in the DOM, or if that fails some sort of regex based pattern matching on the HTML. These are weak and brittle techniques that are pretty much guaranteed to fail at some point.

sentrms 4y ago
I've changed from WordPress to Vue to React but can still use my login e2e test as I depend on a DOM element ID. Eg: #username_field, #password_field, #submit_btn.

felixhummel 4y ago

Using TLS client certificates automatically is still not possible in Playwright [1]. That's our main reason why we keep using Selenium (with pytest-selenium) and Selenium Grid [2]. [1] https://github.com/microsoft/playwright/issues/1799 https://github.com/microsoft/playwright/issues/1799 [2] https://www.selenium.dev/documentation/grid/ https://www.selenium.dev/documentation/grid/

a_c 4y ago

Actually what is stopping you from generating automation selenium script by recording browser action? I recently wrote a tiny browser extension to do just that. Intercept event and translate the action into a selenium script. Currently the supported features are few. But would love to learn what am I expecting down the path.

jawr 4y ago

What’s the best way of monitoring suites? The built in report tool is amazing, but would love a solution that makes for easier analysis and maybe a closer integration with jira.

tmd83 4y ago

Can someone provide an in depth comparison between Cypress and Playwright? It seems that both of them could be good choice compared to Selenium. For example I keep seeing auto-wait being talked about for Playwright but doesn't cypress have that too? It seems playwright has better browser support and multi-window/concurrency story and cypress have better built-in tools (like a test-runner UI). Basically looking for recommendation for test-automation for an old complex app mostly written using server-side tech but has potential for react and such moving forward.

MetaWhirledPeas 4y ago
I haven't used Playwright, but I can tell just by reading about it that the code has more boilerplate and doesn't read as nicely. Example of a Cypress test (pardon the freehand typing here): describe(('the google home page') => { it(('allows you to search for banana pictures') => { cy.visit('https://google.com') cy.get('[aria-label="Search"]').type('banana{enter}') cy.contains('Images').click() cy.get('[alt="Image result for banana"]').should('be.visible') }) }) Nothing too weird, and a lot of the library functions you'd need are available globally without any importing. It even has jQuery and Lodash.
epolanski 4y ago
Playwright supports webkit by default, even in your linux CI. Playwright is faster. It's fully free and OS. It's written by Microsoft to test their own thousands of services not to sell you anything like Cypress. The authors of Playwright are the leads from Puppeteeer, Lighthouse, the first node debugger and Chrome Dev Tools, the engineering team has simply no comparisons in the browser automation field.

lyu07282 4y ago

My journey has been Selenium, then Cypress, then Playwright. This was the first time I was happy writing ui tests, Playwright is great! (Though I don't use the recording tool at all this article focuses on I rather write the tests manually)

baal80spam 4y ago
> Though I don't use the recording tool at all this article focuses on I rather write the tests manually That's because automatic code generators rarely work (anything more dynamically generated and it's a no-go).
- GenerocUsername 4y ago
  I manually do it, then replace parameters where needed
dvngnt_ 4y ago
i miss the cypress test runner
- lyu07282 4y ago
  The thing about cypress that annoyed me the most was how it wasn't just using async/await syntax, generating more complex tests dynamically is super hard. Just try to write a test that follows links on a site dynamically, I couldn't figure it out.
  - btown 4y ago
    Cypress requires developers to have a mental model of "calling a Cypress function actually adds an action to the back of a queue and immediately returns control flow without waiting for that action to run." And this is fine if you can express your waits and conditions declaratively - you give Cypress a plan, it decodes the plan from your Cypress code in milliseconds, and then will take as long as it needs to execute that plan. And Cypress gives a great API surface where almost everything can be done with just this declarative API, and it enables them to show the full plan in a UI even while it's executing. But if you actually need custom control flow, with code you control... you need to "pause" your action-enqueueing to let Cypress catch up. Which means you're going to be nesting callbacks, or sprinkling awaits in all the places that aren't logically awaiting, because you need an explicit yield point out of your JS block. Our team finds this rare enough that it's acceptable. But... yeah, it's far from a perfect solution.

plugin-baby 4y ago

I’ve tried writing e2e tests for web apps using browser drivers on-and-off for the last decade, and it feels like with playwright that the tech has finally matured enough to make the tests reliable enough to be worth the effort of maintaining them.

sh4rks 4y ago
What does Playwright do that Selenium can't do?
- ncrmro 4y ago
  Run inside a container with near zero effort for CI pipelines
- voxic11 4y ago
  The main difference is that Playwright uses custom headless builds of "browser projects" (like chromium) rather than attempting to automate a normal consumer build of the browser via the webdriver api. The webdriver api is kind of slow, browsers don't always implement it well, and it doesn't have a good mechanism for the browser to push events back to the test application which ends up requiring selenium to do a lot of polling for changes so it can miss events that happen quickly.
  - ethbr0 4y ago
    Events are the bane of automation existence. Back in Windows-heavyweight app land, it made way more sense to interpose a low overhead shim to intercept and copy events. I can't imagine the pain you'd encounter trying to paper over the other way around.
    - imiric 4y ago
      > Events are the bane of automation existence. They do make some tasks tricky to handle, like tracking state, or avoiding race conditions, but in reality events are a good fit for interacting with browser DOM events, representing browser activity, and being able to react to it. Consider the complexity of an average modern web site, that loads dozens of scripts, uses a frontend framework, is highly dynamic and interactive (likely a SPA), contains multiple frames, etc. All of this happens concurrently, and the only abstraction that makes sense to represent it is an event-based system. You can't achieve the same level of responsiveness using a traditional request/response protocol. This is why modern browser automation protocols are event-based, and built on top of WebSocket. Selenium is pushing for WebDriver BiDi, while the browser standard is the Chrome DevTools Protocol.

MetaWhirledPeas 4y ago

> So much so, I started looking at Javascript testing frameworks like Chai, Mocha, Cypress etc. The problem I found is that they require a completely different setup, and aren't easy to get started for someone from a Python background. A shame to see Cypress lumped in and dismissed like that. It really is a fantastic way to test. > The killer feature of Playwright is: You can automatically generate tests by opening a web browser and manually running through the steps you want. It saves the hassle I faced with Selenium, where you were opening developer tools and finding the Xpath or similar. Yuck This is absolutely not the primary hassle with testing. Recording steps can help kick start your testing, sure. But very quickly you start to realize that it's only saving you from the easiest work at best, and is creating flaky tests at worst.

salawat 4y ago
Indeed. You need: loggers, data generators, integrations with state probe points, data/result aggregators, etc, etc. For massive numbers of tests, now it's time to look into parallelism. Really hope you're not sharing your test data between test cases! Or holding open a single db session to your write instance! How about multimedia artifacts management? Point being, test automators do a shit ton of work nobody ever wants to dig into. Tests are not there to only be written. They must be read.
sdfhbdf 4y ago
> Cypress lumped in and dismissed like that I think Cypress is really cool when you're on their happy path - testing on Chromium browsers and in JavaScript. I tried it some time ago so maybe this might be outdated, but they had little support for browsers other than Chromium or even rather Electron based and also it was a little difficult to wire up with Gherkin scenarios. Keep in mind that the OP refers to Python tools, AFAIK Cypress does not offer Python bindings while both Selenium and Playwright do.
- jjzhiyuan 4y ago
  We have used Cypress for 3 years to test our oss project: https://apisix.apache.org/blog/2021/02/08/stable-product-delivery-with-cypress/ https://apisix.apache.org/blog/2021/02/08/stable-product-del...

irsagent 4y ago

It must be a game changer as Selenium is mentioned, but never linked in the article.

simonw 4y ago

I just spent half an hour wrestling with Selenium for Python and that API has NOT aged well. content = driver.find_element(By.CSS_SELECTOR, "p.content") In Playwright Python: content = page.locator("p.content") Playwright is just nicer to use. Selenium feels very Java-ish.

KyeRussell 4y ago
Yeah, not only that, but that’s the NEWER Selenium Python API. The older style was actually (imo) easier to grapple with, discover, etc. The whole thing has felt like it’s either written by people that don’t work with Python very often, or that it’s too married to the idea of being a close port of an API targeted at…yeah…Java.