4 ms·

Here's a tricky one that I encountered a year ago. A junior tried to advocate for browser test. Senior from the developer productivity team said browser test c

Here's a tricky one that I encountered a year ago. A junior tried to advocate for browser test. Senior from the developer productivity team said browser test couldn't possibly be made non-flaky. I didn't know how to advice the junior because I agreed with them. Saying "something is infeasible/costly" is like a blanket statement. There was no way to quantify that. On a flip side, we couldn't justify the impact of browser test either. But I felt, at the time, we should've been biased to implement every type of tests (than not) since there are only 3 types: backend, JS, and browser. Akin to your example, it was the same argument "X is costly/infeasible". Your example doesn't have any issue because everyone probably agrees with it. But being infeasible to setup browser test sounds strange. Also, if we don't plant the tree now, then when?

cousin_it 7y ago
There's another type of tests: visual diffing. I've used it a lot and I love it. 1) Keep a set of a few thousand URLs to diff, add new ones as needed. 2) Use a tool that can request these URLs from two versions of your app, make a side-by-side visual diff of the resulting pages, and present a report of any differences. 3) Before committing any change to your codebase, run that tool automatically on the whole URL set, comparing the new version of the app with the current version. Then the committer should review the diff report manually and it gets attached to the commit history. This way, adding coverage for new functionality is easy (you just add some URLs to a text file) and the whole thing runs fast. And it catches all kinds of problems. Not just UI bugs where some bit of CSS messes up something unrelated, but also you can run the frontend diff after making any backend change and it will catch problems as well. It won't solve all your testing needs, but it covers a lot of ground cheaply, so you can concentrate the custom testing code where it's actually needed.
maliker 7y ago
My team went through the browser testing issue as well and agree they're hard tests to write. We ended up writing a few using selenium, but we don't have as much coverage as we'd like. Luckily, unlike an interpreter upgrade, we can add them incrementally.
- ergocoder 7y ago
  This is the way it should be. If the test is slow and hard to deflake, then let's add it slowly. Maybe only test the critical path. There are ways to manage it, instead of discarding it entirely.

cousin_it 7y ago

There's another type of tests: visual diffing. I've used it a lot and I love it. 1) Keep a set of a few thousand URLs to diff, add new ones as needed. 2) Use a tool that can request these URLs from two versions of your app, make a side-by-side visual diff of the resulting pages, and present a report of any differences. 3) Before committing any change to your codebase, run that tool automatically on the whole URL set, comparing the new version of the app with the current version. Then the committer should review the diff report manually and it gets attached to the commit history. This way, adding coverage for new functionality is easy (you just add some URLs to a text file) and the whole thing runs fast. And it catches all kinds of problems. Not just UI bugs where some bit of CSS messes up something unrelated, but also you can run the frontend diff after making any backend change and it will catch problems as well. It won't solve all your testing needs, but it covers a lot of ground cheaply, so you can concentrate the custom testing code where it's actually needed.

maliker 7y ago

My team went through the browser testing issue as well and agree they're hard tests to write. We ended up writing a few using selenium, but we don't have as much coverage as we'd like. Luckily, unlike an interpreter upgrade, we can add them incrementally.

ergocoder 7y ago
This is the way it should be. If the test is slow and hard to deflake, then let's add it slowly. Maybe only test the critical path. There are ways to manage it, instead of discarding it entirely.