16 ms·
They already do that with Firefox Nightly. But most users expect a more stable product than that.
by uranusjr 7y ago
They already do that with Firefox Nightly. But most users expect a more stable product than that.
- chrisseaton 7y agoHow can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?
- pingyong 7y agoIt can be more stable by having nightly users test it beforehand for a while and report any new bugs.
- chrisseaton 7y agoWhat are those users doing that can’t be automated and run in CI?
- chopin 7y agoTest on exotic hardware, for example. Or exotic combinations of libraries in the OS which FF depends on.
- chrisseaton 7y agoThose should be in the CI test matrix then - we shouldn’t be relying on manual testing done specially for releases.
- chopin 7y agoIf even Microsoft can't catch these, why should Mozilla be able to? I am admittedly not in the CI business for frontends but I'd imagine that the combinatorics vastly outnumbers the possibilities of any organization.
- pingyong 7y agoI'm guessing something the devs haven't thought of. As soon as you hit 10k+ users, chances are people are doing things you didn't expect.
- atroche 7y agoIt'd be pretty ambitious to have CI run against the variety of web apps that real users "test" against (including ones they're developing!)
- ashwoods 7y agoTo begin with, it is probably released with a subset of the CI matrix, because I imagine that the full CI matrix takes more than a day. Second, it is nearly impossible* to cover all possible environments+addons+websites and test them correctly. This isn't a normal desktop app, nor a webapp that always is running relatively sandboxed in a more or less known environment. Browsers are platforms, as OS are. Lots of people doing wacky stuff on it. Very hard to realistically automate all that stuff. *impossible with a limited constraint of time and/or money.