4 ms·
They must already be doing a full build, test, and benchmark for each platform for each PR to their codebase in CI, presumably in parallel to make it run reason
by chrisseaton 7y ago
They must already be doing a full build, test, and benchmark for each platform for each PR to their codebase in CI, presumably in parallel to make it run reasonably fast. It doesn’t seem like much of an extra step to release those binaries they’re already building hundreds of times a day.
- uranusjr 7y agoThey already do that with Firefox Nightly. But most users expect a more stable product than that.
- chrisseaton 7y agoHow can it be more stable than a build that passes CI? If they have some extra tests to run to check it’s stable why aren’t those in CI?
- pingyong 7y agoIt can be more stable by having nightly users test it beforehand for a while and report any new bugs.
- chrisseaton 7y agoWhat are those users doing that can’t be automated and run in CI?
- chopin 7y agoTest on exotic hardware, for example. Or exotic combinations of libraries in the OS which FF depends on.
- chrisseaton 7y agoThose should be in the CI test matrix then - we shouldn’t be relying on manual testing done specially for releases.
- chopin 7y agoIf even Microsoft can't catch these, why should Mozilla be able to? I am admittedly not in the CI business for frontends but I'd imagine that the combinatorics vastly outnumbers the possibilities of any organization.
- pingyong 7y agoI'm guessing something the devs haven't thought of. As soon as you hit 10k+ users, chances are people are doing things you didn't expect.
- atroche 7y agoIt'd be pretty ambitious to have CI run against the variety of web apps that real users "test" against (including ones they're developing!)
- ashwoods 7y agoTo begin with, it is probably released with a subset of the CI matrix, because I imagine that the full CI matrix takes more than a day. Second, it is nearly impossible* to cover all possible environments+addons+websites and test them correctly. This isn't a normal desktop app, nor a webapp that always is running relatively sandboxed in a more or less known environment. Browsers are platforms, as OS are. Lots of people doing wacky stuff on it. Very hard to realistically automate all that stuff. *impossible with a limited constraint of time and/or money.
- padenot 7y agoWe do exactly this for people who want. A reasonably stable build (including full testing, and benchmarking, which takes multiple hundreds hours, parallelized of course) is available twice a day (0:00 and 12:00 UTC). It self updates (either differential or full download depending of what landed on the past day). I've been using only this for the last 10 years or so (on windows, Linux, Mac and Android), and I can count the number of time things really broke on one hand. We call these Firefox Nighly, and they are available at https://www.mozilla.org/en-US/firefox/nightly/all/ https://www.mozilla.org/en-US/firefox/nightly/all/. Release notes at available at https://www.mozilla.org/en-US/firefox/71.0a1/releasenotes/ https://www.mozilla.org/en-US/firefox/71.0a1/releasenotes/, but they are not always updated for each version, sometimes batched (its quite some work due to the amount of commits that are pushed per day). There is a cool blog at https://blog.nightly.mozilla.org/ https://blog.nightly.mozilla.org/ that talks a bit about the new stuff. All this is really experimental and probably not for everybody, but helps us to deliver something good at the end. Filing bugs and giving us feedback is of course particularly appreciated, as always!
- fluffything 7y agoAre nightlies tested on CI for all platforms that Firefox supports ? I only see Tier-1 in that matrix (Linux, Windows, Mac). No iOS, Android, *BSDs, etc.