6 ms·
How can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?
by chrisseaton 7y ago
How can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?
- pingyong 7y agoIt can be more stable by having nightly users test it beforehand for a while and report any new bugs.
- chrisseaton 7y agoWhat are those users doing that can’t be automated and run in CI?
- chopin 7y agoTest on exotic hardware, for example. Or exotic combinations of libraries in the OS which FF depends on.
- chrisseaton 7y agoThose should be in the CI test matrix then - we shouldn’t be relying on manual testing done specially for releases.
- chopin 7y agoIf even Microsoft can't catch these, why should Mozilla be able to? I am admittedly not in the CI business for frontends but I'd imagine that the combinatorics vastly outnumbers the possibilities of any organization.
- pingyong 7y agoI'm guessing something the devs haven't thought of. As soon as you hit 10k+ users, chances are people are doing things you didn't expect.
- atroche 7y agoIt'd be pretty ambitious to have CI run against the variety of web apps that real users "test" against (including ones they're developing!)
- ashwoods 7y agoTo begin with, it is probably released with a subset of the CI matrix, because I imagine that the full CI matrix takes more than a day. Second, it is nearly impossible* to cover all possible environments+addons+websites and test them correctly. This isn't a normal desktop app, nor a webapp that always is running relatively sandboxed in a more or less known environment. Browsers are platforms, as OS are. Lots of people doing wacky stuff on it. Very hard to realistically automate all that stuff. *impossible with a limited constraint of time and/or money.