3 ms·

I've largely moved away from UI testing. The two main reasons are: 1) Flakiness / non-determinism 2) The constant change of the UI Both of these are absolute

I've largely moved away from UI testing. The two main reasons are: 1) Flakiness / non-determinism 2) The constant change of the UI Both of these are absolute killers to productivity on a large team. Note, I'm not against UI testing in theory. I think if you could, you would have full end to end tests for every single edge case so that you could be sure that the actual product presented to users work. But in practice, end to end testing an asynchronous distributed system (which a simple client-server application still is) is full of non-determinism. Re the constant changing of the UI. This is just also true in my experience. I've worked on a navigation / global UI redesign at every company I've ever worked at. It happens like once every 3-5 years. Within the redesigns, it's still extremely common to subtly change the UI for UX reasons all the time. When this happens, be prepared to spend half of the time updating all of your tests.

inglor 6y ago
To be fair, I work for a company in this space (Testim.io) and we have had success stories with very large companies doing UI tests with the service (like Microsoft). I think the hardest aspects of testing are fast authoring and stability (maintenance) - AI tools can help with that and learning data between test runs can create very stable tests. So as someone in this space - it's a very exciting space and I am very optimistic for this startup (Reflect)
- amw-zero 6y ago
  Yea I don't want to discourage work in the space. Actually, quite the opposite. I'm just saying where the bar is for me when I'm considering handing over money. I'm not going to pay money for something that ends up actually slowing me down.
hinkley 6y ago
First AJAX heavy app I worked on, we rigged the game so that we could win. We wired up all of the asynchronous bits to tweak a hidden DOM element at the bottom of the page, so that our tests could wait for that element to settle before validating anything. We'd already had a lot of our async code run through a couple of helper methods (to facilitate cache busting and Karma tests), so it was 'just' a matter of finishing what we'd started. I kinda feel like the browsers are letting us down by not exposing this sort of stuff.
- atarian 6y ago
  How does that address the issue of the UI changing? If the markup changed, you still need to update your tests.
  - hinkley 6y ago
    A great deal of your non-determinism is in trying to run an End-to-End test while the UI is still trying to settle. For instance, your negative tests (click something and make sure nothing happens) can fail to detect anything at all because they perform an action and then validate the UI before the action has even processed. Yes, of course nothing has happened, because you didn't let it.
    - amw-zero 6y ago
      Let me clarify. Non-determinism is just one issue. The other issue is how frequently the UI changes. For example, a large form can get broken out into a wizard. The UI is completely semantically different, but performs the same functionality. How do you make such UI changes without your test suite slowing you down?
kmclean 6y ago
This sounds pretty similar to my experience with UI testing. What kinds of tests do you write instead? And how have they worked out for your team?
- amw-zero 6y ago
  We tend to create a "view model" object which maintains the state of the UI and interacts with the server. The actual view just forwards commands to this object, and after a command happens the view always re-renders based on the current state of the model. We do this on iOS but I've been pushing for us to do it with React as well since you get the built-in state rendering for free. So all of the tests look something like this (here's a UI form): // Arrange let searchResponse = { usernameTaken: false }; let networkService = NetworkServiceTestDouble(searchResponse); let vm = UserFormViewModel( networkService: networkService ); // Act vm.username = "newusername@email.com"; vm.checkUsernameAvailable(); vm.submitForm(); // Assert expect(networkService.calls[0].arguments[0]).to.eq({ username: "newusername@email.com" }); So you see the view model is kind of like a Page Object, in that it has an API which represents the user's interaction. But, in this case the view model is what's used in the actual production code, it's not just a testing helper. So you just invoke the API as the user would when interacting with the feature, and verify that the correct data gets sent to the server when the form is submitted. After all, the client's job is primarily to interact with the server and maintain state. The feature is functional if the correct data is sent to the server. Then, let's say you're in React, hooking up this view model is quite easy: import { UserFormViewModel } from "./UserFormViewModel"; let vm = UserFormViewModel(realNetworkService); function UserForm() { return ( <button onClick={vm.submitForm()}>Submit</button> ) } The buttons / inputs just get wired up to the appropriate view model functions and properties, and you can have the component's state just be the view model's state. The view is then just entirely about appearance and forwarding events. I'll then write some tests that actually render the UI, but they're just sanity checks. How often do you get a bug where you failed to call the correct function in response to a button click? The logic and wiring in response to all of the events is the more interesting part. Edited to improve the code formatting a bit.
  - 6y ago

inglor 6y ago

To be fair, I work for a company in this space (Testim.io) and we have had success stories with very large companies doing UI tests with the service (like Microsoft). I think the hardest aspects of testing are fast authoring and stability (maintenance) - AI tools can help with that and learning data between test runs can create very stable tests. So as someone in this space - it's a very exciting space and I am very optimistic for this startup (Reflect)

amw-zero 6y ago
Yea I don't want to discourage work in the space. Actually, quite the opposite. I'm just saying where the bar is for me when I'm considering handing over money. I'm not going to pay money for something that ends up actually slowing me down.

hinkley 6y ago

First AJAX heavy app I worked on, we rigged the game so that we could win. We wired up all of the asynchronous bits to tweak a hidden DOM element at the bottom of the page, so that our tests could wait for that element to settle before validating anything. We'd already had a lot of our async code run through a couple of helper methods (to facilitate cache busting and Karma tests), so it was 'just' a matter of finishing what we'd started. I kinda feel like the browsers are letting us down by not exposing this sort of stuff.

atarian 6y ago
How does that address the issue of the UI changing? If the markup changed, you still need to update your tests.
- hinkley 6y ago
  A great deal of your non-determinism is in trying to run an End-to-End test while the UI is still trying to settle. For instance, your negative tests (click something and make sure nothing happens) can fail to detect anything at all because they perform an action and then validate the UI before the action has even processed. Yes, of course nothing has happened, because you didn't let it.
  - amw-zero 6y ago
    Let me clarify. Non-determinism is just one issue. The other issue is how frequently the UI changes. For example, a large form can get broken out into a wizard. The UI is completely semantically different, but performs the same functionality. How do you make such UI changes without your test suite slowing you down?

kmclean 6y ago

This sounds pretty similar to my experience with UI testing. What kinds of tests do you write instead? And how have they worked out for your team?

amw-zero 6y ago
We tend to create a "view model" object which maintains the state of the UI and interacts with the server. The actual view just forwards commands to this object, and after a command happens the view always re-renders based on the current state of the model. We do this on iOS but I've been pushing for us to do it with React as well since you get the built-in state rendering for free. So all of the tests look something like this (here's a UI form): // Arrange let searchResponse = { usernameTaken: false }; let networkService = NetworkServiceTestDouble(searchResponse); let vm = UserFormViewModel( networkService: networkService ); // Act vm.username = "newusername@email.com"; vm.checkUsernameAvailable(); vm.submitForm(); // Assert expect(networkService.calls[0].arguments[0]).to.eq({ username: "newusername@email.com" }); So you see the view model is kind of like a Page Object, in that it has an API which represents the user's interaction. But, in this case the view model is what's used in the actual production code, it's not just a testing helper. So you just invoke the API as the user would when interacting with the feature, and verify that the correct data gets sent to the server when the form is submitted. After all, the client's job is primarily to interact with the server and maintain state. The feature is functional if the correct data is sent to the server. Then, let's say you're in React, hooking up this view model is quite easy: import { UserFormViewModel } from "./UserFormViewModel"; let vm = UserFormViewModel(realNetworkService); function UserForm() { return ( <button onClick={vm.submitForm()}>Submit</button> ) } The buttons / inputs just get wired up to the appropriate view model functions and properties, and you can have the component's state just be the view model's state. The view is then just entirely about appearance and forwarding events. I'll then write some tests that actually render the UI, but they're just sanity checks. How often do you get a bug where you failed to call the correct function in response to a button click? The logic and wiring in response to all of the events is the more interesting part. Edited to improve the code formatting a bit.
- 6y ago