I do a similar style of UI test automation for set-top boxes / smart TVs, with stb-tester[1]. We've found that the "Got kicked off for every build" continuous

I do a similar style of UI test automation for set-top boxes / smart TVs, with stb-tester[1]. We've found that the "Got kicked off for every build" continuous integration process you mention is the crucial part to achieving success with this type of test automation -- if you're going to invest the effort in writing reliable tests, you want to be getting value out of them by running them as often as possible and as early as possible. [1] http://stb-tester.com/ http://stb-tester.com/