3 ms·

> And remember, nothing was broken in a way that a machine would notice, so automated acceptance tests wouldn't have caught this regression. I'd like to nit he

by tvararu 11y ago

> And remember, nothing was broken in a way that a machine would notice, so automated acceptance tests wouldn't have caught this regression. I'd like to nit here that visual regression test harnesses for web application development do exist. [1] [2] [3] Using something like Sauce Labs you could conceptually have these tests run on Windows machines in the cloud. However, having implemented visual regression testing in the past in various projects, I can personally vouch that it's a huge waste of time. Your tools will break all the time, your tests will break all the time, and you'll generally have a bad time. [1] https://github.com/chenglou/node-huxley https://github.com/chenglou/node-huxley [2] https://github.com/Huddle/PhantomCSS https://github.com/Huddle/PhantomCSS [3] https://github.com/bradgignac/janus https://github.com/bradgignac/janus

jwmerrill 11y ago
Thanks for this perspective. I've never tried this kind of automated visual regression testing except in very specialized circumstances, but I've sometimes thought that I should. In analogy with scientific modeling, there's a risk of over-fitting your tests against the exact circumstances of your product as it exists today. You want tests to capture the important aspects of your product, while having a little bit of flexibility for the "noise" related to differences between platforms, and changes to mostly but not entirely unrelated parts of the product.
- tvararu 11y ago
  The first time I did visual regression testing, I generated a bunch of screenshots on my Mac, chucked them into version control, and then ran the tests on CI. They broke because of the font rendering differences between OS X and Ubuntu. The tool I used, PhantomCSS, did allow for a threshold underneath which pixel differences would not get picked up, but I opted instead to run my tests locally in vagrant so that the environment would be closer to identical to CI. So yes, platform noise is just one of the myriad of challenges you have to solve when implementing such tooling.

by tvararu 11y ago

jwmerrill 11y ago

Thanks for this perspective. I've never tried this kind of automated visual regression testing except in very specialized circumstances, but I've sometimes thought that I should. In analogy with scientific modeling, there's a risk of over-fitting your tests against the exact circumstances of your product as it exists today. You want tests to capture the important aspects of your product, while having a little bit of flexibility for the "noise" related to differences between platforms, and changes to mostly but not entirely unrelated parts of the product.

tvararu 11y ago
The first time I did visual regression testing, I generated a bunch of screenshots on my Mac, chucked them into version control, and then ran the tests on CI. They broke because of the font rendering differences between OS X and Ubuntu. The tool I used, PhantomCSS, did allow for a threshold underneath which pixel differences would not get picked up, but I opted instead to run my tests locally in vagrant so that the environment would be closer to identical to CI. So yes, platform noise is just one of the myriad of challenges you have to solve when implementing such tooling.