13 ms·

I've always wondered this: why release on a schedule at all? If people demand a certain new feature ("[this will] bring you new features more quickly"), it can

by lucb1e 7y ago

I've always wondered this: why release on a schedule at all? If people demand a certain new feature ("[this will] bring you new features more quickly"), it can just be brought out as it's ready, instead of having to wait an average of 3 (now 2) weeks before it can be released? And why does this have to increment the major version, since when is every single update backwards incompatible?

kelnos 7y ago
I think it's more a matter of discipline than anything else. It's easy to get lost in the weeds with a ton of things waiting to be merged, and over and over saying "just one more PR to merge and then I'll release", and then it's a year later and you still haven't released. Getting features out to people faster means shorter feedback cycles. Regarding versioning, most applications don't use semantic versioning; that's mainly for libraries. Not sure what "backwards incompatible" would mean for a browser, anyway.
- Karunamon 7y ago
  Addons, plug-ins, CSS and HTML directives, etc.
  - dikei 7y ago
    I would argue that you should do features detection rather than depending on version number, as much as possible.
kenhwang 7y ago
Firefox probably had bad memories of when they "released when it's ready" during the FF3 -> FF4 days where a couple of features held back everything. With fix released cycles, they have more of a motivator to cut off things that aren't ready yet so that the things that are can go out.
- jaas 7y ago
  "probably had bad memories" Speaking as someone who was at Mozilla at that time, you can change "probably had" to "definitely has". In hindsight I'd go ahead and call it a disaster. The X weeks release cadence solves so many problems.
  - fetbaffe 7y ago
    Remember back then when enterprises certified their entire install base against FF3, but then FF4 came out & they needed to redo all the enterprise certification bullshit. zach_galifianakis_laugh.mp4
- cpeterso 7y ago
  With shorter release cycles, there's also less pressure to ship half-baked features. If you know the next release is just four weeks away, then slipping a feature one release is rarely a huge deal. But if the next release is one quarter or one year away, teams rush to ship now and fix bugs in production later. Overall product quality will take a hit at every release.
  - zbraniecki 7y ago
    Another thing that I observed in our culture since we switched is that it is easier to coordinate cross-team efforts - if I know X will be done in Fx70, I can plan my work on top of it for 71. Such logic is commonly performed without releases until you start depending on multiple features in multiple places. Then it saves you. The same "alignment" works also much better between engineering and non-engineering teams. Once we collect the focus areas and major features planned for release X, security, performance, accessibility, localization, platform integration, UI, UX, design, legal and marketing teams can prioritize their work to aid them. Having no release cycle, or long release cycle, dilutes this effort and reduces visibility. There is, in my opinion, somewhere, a threshold beyond which the release cycle burden outweighs the benefits and/or where the cycle is to short for teams to usefully react, but I don't think we hit it so far.
  - wpietri 7y ago
Zenst 7y ago
Good point, but factor in build/release, then deployment. We all know what some companies are like in their desktop real estate rollouts - if you have too frequent updates, you end up with greater disparity in releases with a template support of - oh that's not the latest, pls try again. Then feedback, if you rollout release too frequently you also can be several releases in and get feedback of some bug in a release prior to current release, albeit only a week or so. So by having organised releases, allows better management and a better experience overall. It's a fine balance, but for those who needs it, there will always be nightly builds of most things they want with last second updates, also self-builds for those inclined. But I get the whole artificial timeframe does seem somewhat odd, but the further you look at the overall usage, management and support factors et all, you start to appreciate such regular timeframes. On the other extreme, I've worked in an environment in which a whole suit had to be retested and go thru all the hopes as a full stop was missed of a comment and that change caused the whole testing, management/release cycle to go thru the motions again. That you would like, ignore and package up with other small things into a full release - which is what many do these days in most environments. After all - would you want to go thru a rollout of a new release that may contain a fix you want, or equally may just be adding a full stop to a comment in the code and yielding no binary level changes, albeit that it went thru the whole hoops and hurdles as a full release and that is what you got. So from my experience, a 4 week release cycle is good, fixed, something that you can plan around. Also kudos for not just picking a day a month and doing monthly updates like most, that tends to seem more arbitrary unlike 4 weeks, which has an air of less arbitrary and feels like it was thought out more. Though an element of personal experience does come into play.
- cpeterso 7y ago
  > kudos for not just picking a day a month and doing monthly updates like most, that tends to seem more arbitrary unlike 4 weeks Planning feature development and rollout is more difficult if you always have to refer to a calendar to find future release dates. I would prefer to be able to say a feature can land in Firefox Nightly in January, ship to Firefox Beta on February 1, and Release channel users on March 1. Mozilla has a long wiki page of past and future release dates: https://wiki.mozilla.org/RapidRelease/Calendar https://wiki.mozilla.org/RapidRelease/Calendar
yborg 7y ago
>why release on a schedule at all Exactly my thought - why 4 weeks? Why not just put everyone on Nightly and be done with it? Then everyone will enjoy the firehose of agility and features that is Mozilla! You're always shipping working code, right? If it passes the harness, it's good, isn't it?
fluffything 7y ago
Firefox ships ~100 binaries for 100 different targets, from mobile phones, to desktops, tablets, ... x86, arm, sparc, ppc, riscv, ... X linux, windows, macos, freebds, openbsd, netbsd, .... X arch linux, debian, ubuntu .. At some point you need to say this is what we want to ship, and branch, and make sure that not only all tests pass, but that the installers work, run the benchmark suite that might not be run on every PR, make sure the updates work on all systems, appstores, etc. Accept new patches to fix bugs, etc. I suppose they might have a release team that does nothing but releases, and if creating a full release takes a week (no idea how long it takes honestly), then maybe they could release each week. But I don't think they can release a new version every day, because probably running their whole CI takes longer. E.g. before shipping a version of Rust, the RC is used to build and run tests of all packages in the Rust package repository. This takes 4 full days. One could probably throw money at the problem and speed it up, but at least right now, releases just cannot technically happen faster, unless you are willing to compromise on quality.
- aquabeagle 7y ago
  Mozilla only provides binaries for Linux 32-bit, Linux 64-bit, macOS, win32, and win64: http://ftp.mozilla.org/pub/firefox/releases/69.0/ http://ftp.mozilla.org/pub/firefox/releases/69.0/ All of the various Linux distributions just repackage those binaries. iOS and Android are distributed only through the App Store and Google Play. The BSDs and other ("tier 3") platforms have to compile from source and distribute their own packages: https://developer.mozilla.org/en-US/docs/Mozilla/Supported_build_configurations https://developer.mozilla.org/en-US/docs/Mozilla/Supported_b...
  - acqq 7y ago
    > All of the various Linux distributions just repackage those binaries Debian and Ubuntu don’t build from sources?
    - throwaway170919 7y ago
      They do. You can notice by looking at the build time in the version string. This makes it impossible to share a profile between two GNU/Linux distributions (or even the official binary and the distribution binary) without hacking with the current versions, even if they are effectively the same version: one will complain that the profile has been run with a more recent version. (But I sure expect and want my distributions to rebuild Firefox - I trust Debian for making sure no bit of proprietary code is here, and makes tracking opt-in, more than Mozilla)
bytematic 7y ago
At my work the culture is just to ship changes as soon as they improve the product/codebase
- re4ctor 7y ago
  Doing that for a binary release that is in dozens of platforms, configurations, locales, etc. is incredibly time consuming. Uptake on binaries is also usually on the order of days-weeks before most of your user base is up to date. It's easier for say a website or service that can deploy to millions of users within minutes.
  - bytematic 7y ago
    Yeah, I agree for sure. I get tons of updates for spotify, discord, and other applications that don't need to do complex binary distribution
therealjumbo 7y ago
This is a great overview of different release management strategies and the practical implications: https://www.youtube.com/watch?v=IKsQsxubuAA https://www.youtube.com/watch?v=IKsQsxubuAA Not part of the talk: if you have more than 3 people involved you probably need a release schedule in terms of either scope or time. I'd personally advise you to have one even if you only have 1 person, just for a mental target if nothing else. The talk goes into studies of projects which did scope based releases and got horrible results in practice. All the ones in the talk switched to time based releases and were much happier. Also, in a software project, you can only fix either scope or time, but not both. Also, if you want to ship in a timely manner, you probably want to fix one of them, not let both float. Edit: it looks like this is his paper on the topic: http://www.cyrius.com/publications/michlmayr_fitzgerald-time_based_release_management.pdf http://www.cyrius.com/publications/michlmayr_fitzgerald-time...
aylmao 7y ago
> Why release on a schedule at all? I feel like it might help with motivation. It's easy to push things "to tomorrow" if you don't have a hard cap on how many tomorrows are available before the feature is supposed to go out. > If people demand a certain new feature ("[this will] bring you new features more quickly"), it can just be brought out as it's ready, instead of having to wait an average of 3 (now 2) weeks before it can be released? It's kind of like forming a habit. Having a set schedule could in theory delay quick features, but it will also keep the motivation going for time-consuming features. > And why does this have to increment the major version, since when is every single update backwards incompatible? I actually really like this personally. Encoding compatibility in a version number is, IMO, useless— it's sometimes just hard to know if your changes will break something or not, so there's a wealth of minor versions out there that break things anyway. Just using a single number just makes everything simpler. Detecting breaking changes should be done with systems (tests, static analysis, etc) and not by trusting a number some developer came up with.
deleted 7y ago
[deleted]
batmansmk 7y ago
For quite a few products, releasing takes time. QA, auditing the security, localizing, generating the builds, executing exhaustive in-depth unit tests, flushing caches, writing release notes, publicating on different websites... it can take a while! When the cost of the release goes above a certain amount, due the sheer number of features or the thoroughness of the release process, it starts to make sense to mutualize release tasks between a bunch of features. Bundling those features on a time basis is quite often a local optimum between an acceptable delay for the product team and an acceptable cost for ops/release/qa/...
wpietri 7y ago
They're clearly moving gradually in this direction, but I think it's an open question how far they can get. For server-side software, it's easy enough to do multiple releases per day, because you have full control over the environment, and all the work is internal. But for desktop/mobile software like this, having multiple releases per day won't be efficient in terms of bandwidth, load, or user experience. And each browser version has other external costs, like documentation and support. Right now it can be hard to solve a problem like, "How do I do X with Firefox" because the answer varies by version. The more versions you have, the harder that is. Or can you imagine trying to triage and verify bug reports if there are, say, 3 new Firefox releases per day, and people could be on any one of dozens of versions because they don't quit and restart the browser very often? We can imagine solutions to these problems, of course. But I think getting a release cadence below weekly for desktop apps will be a very long road.
roca 7y ago
One issue with releasing every day or every hour is that that would mean users downloading massive numbers of updates. Sure, you ship deltas, but there's still a cost to each update.
Thorrez 7y ago
Probably for a similar reason to why c++ releases a new standard every 3 years. This blog post explains c++'s reasoning: https://herbsutter.com/2019/07/13/draft-faq-why-does-the-c-standard-ship-every-three-years/ https://herbsutter.com/2019/07/13/draft-faq-why-does-the-c-s...

by lucb1e 7y ago

kelnos 7y ago

I think it's more a matter of discipline than anything else. It's easy to get lost in the weeds with a ton of things waiting to be merged, and over and over saying "just one more PR to merge and then I'll release", and then it's a year later and you still haven't released. Getting features out to people faster means shorter feedback cycles. Regarding versioning, most applications don't use semantic versioning; that's mainly for libraries. Not sure what "backwards incompatible" would mean for a browser, anyway.

Karunamon 7y ago
Addons, plug-ins, CSS and HTML directives, etc.
- dikei 7y ago
  I would argue that you should do features detection rather than depending on version number, as much as possible.

kenhwang 7y ago

Firefox probably had bad memories of when they "released when it's ready" during the FF3 -> FF4 days where a couple of features held back everything. With fix released cycles, they have more of a motivator to cut off things that aren't ready yet so that the things that are can go out.

jaas 7y ago
"probably had bad memories" Speaking as someone who was at Mozilla at that time, you can change "probably had" to "definitely has". In hindsight I'd go ahead and call it a disaster. The X weeks release cadence solves so many problems.
- fetbaffe 7y ago
  Remember back then when enterprises certified their entire install base against FF3, but then FF4 came out & they needed to redo all the enterprise certification bullshit. zach_galifianakis_laugh.mp4
cpeterso 7y ago
With shorter release cycles, there's also less pressure to ship half-baked features. If you know the next release is just four weeks away, then slipping a feature one release is rarely a huge deal. But if the next release is one quarter or one year away, teams rush to ship now and fix bugs in production later. Overall product quality will take a hit at every release.
- zbraniecki 7y ago
  Another thing that I observed in our culture since we switched is that it is easier to coordinate cross-team efforts - if I know X will be done in Fx70, I can plan my work on top of it for 71. Such logic is commonly performed without releases until you start depending on multiple features in multiple places. Then it saves you. The same "alignment" works also much better between engineering and non-engineering teams. Once we collect the focus areas and major features planned for release X, security, performance, accessibility, localization, platform integration, UI, UX, design, legal and marketing teams can prioritize their work to aid them. Having no release cycle, or long release cycle, dilutes this effort and reduces visibility. There is, in my opinion, somewhere, a threshold beyond which the release cycle burden outweighs the benefits and/or where the cycle is to short for teams to usefully react, but I don't think we hit it so far.
- wpietri 7y ago

Zenst 7y ago

Good point, but factor in build/release, then deployment. We all know what some companies are like in their desktop real estate rollouts - if you have too frequent updates, you end up with greater disparity in releases with a template support of - oh that's not the latest, pls try again. Then feedback, if you rollout release too frequently you also can be several releases in and get feedback of some bug in a release prior to current release, albeit only a week or so. So by having organised releases, allows better management and a better experience overall. It's a fine balance, but for those who needs it, there will always be nightly builds of most things they want with last second updates, also self-builds for those inclined. But I get the whole artificial timeframe does seem somewhat odd, but the further you look at the overall usage, management and support factors et all, you start to appreciate such regular timeframes. On the other extreme, I've worked in an environment in which a whole suit had to be retested and go thru all the hopes as a full stop was missed of a comment and that change caused the whole testing, management/release cycle to go thru the motions again. That you would like, ignore and package up with other small things into a full release - which is what many do these days in most environments. After all - would you want to go thru a rollout of a new release that may contain a fix you want, or equally may just be adding a full stop to a comment in the code and yielding no binary level changes, albeit that it went thru the whole hoops and hurdles as a full release and that is what you got. So from my experience, a 4 week release cycle is good, fixed, something that you can plan around. Also kudos for not just picking a day a month and doing monthly updates like most, that tends to seem more arbitrary unlike 4 weeks, which has an air of less arbitrary and feels like it was thought out more. Though an element of personal experience does come into play.

cpeterso 7y ago
> kudos for not just picking a day a month and doing monthly updates like most, that tends to seem more arbitrary unlike 4 weeks Planning feature development and rollout is more difficult if you always have to refer to a calendar to find future release dates. I would prefer to be able to say a feature can land in Firefox Nightly in January, ship to Firefox Beta on February 1, and Release channel users on March 1. Mozilla has a long wiki page of past and future release dates: https://wiki.mozilla.org/RapidRelease/Calendar https://wiki.mozilla.org/RapidRelease/Calendar

yborg 7y ago

>why release on a schedule at all Exactly my thought - why 4 weeks? Why not just put everyone on Nightly and be done with it? Then everyone will enjoy the firehose of agility and features that is Mozilla! You're always shipping working code, right? If it passes the harness, it's good, isn't it?

fluffything 7y ago

Firefox ships ~100 binaries for 100 different targets, from mobile phones, to desktops, tablets, ... x86, arm, sparc, ppc, riscv, ... X linux, windows, macos, freebds, openbsd, netbsd, .... X arch linux, debian, ubuntu .. At some point you need to say this is what we want to ship, and branch, and make sure that not only all tests pass, but that the installers work, run the benchmark suite that might not be run on every PR, make sure the updates work on all systems, appstores, etc. Accept new patches to fix bugs, etc. I suppose they might have a release team that does nothing but releases, and if creating a full release takes a week (no idea how long it takes honestly), then maybe they could release each week. But I don't think they can release a new version every day, because probably running their whole CI takes longer. E.g. before shipping a version of Rust, the RC is used to build and run tests of all packages in the Rust package repository. This takes 4 full days. One could probably throw money at the problem and speed it up, but at least right now, releases just cannot technically happen faster, unless you are willing to compromise on quality.

aquabeagle 7y ago
Mozilla only provides binaries for Linux 32-bit, Linux 64-bit, macOS, win32, and win64: http://ftp.mozilla.org/pub/firefox/releases/69.0/ http://ftp.mozilla.org/pub/firefox/releases/69.0/ All of the various Linux distributions just repackage those binaries. iOS and Android are distributed only through the App Store and Google Play. The BSDs and other ("tier 3") platforms have to compile from source and distribute their own packages: https://developer.mozilla.org/en-US/docs/Mozilla/Supported_build_configurations https://developer.mozilla.org/en-US/docs/Mozilla/Supported_b...
- acqq 7y ago
  > All of the various Linux distributions just repackage those binaries Debian and Ubuntu don’t build from sources?
  - throwaway170919 7y ago
    They do. You can notice by looking at the build time in the version string. This makes it impossible to share a profile between two GNU/Linux distributions (or even the official binary and the distribution binary) without hacking with the current versions, even if they are effectively the same version: one will complain that the profile has been run with a more recent version. (But I sure expect and want my distributions to rebuild Firefox - I trust Debian for making sure no bit of proprietary code is here, and makes tracking opt-in, more than Mozilla)

bytematic 7y ago

At my work the culture is just to ship changes as soon as they improve the product/codebase

re4ctor 7y ago
Doing that for a binary release that is in dozens of platforms, configurations, locales, etc. is incredibly time consuming. Uptake on binaries is also usually on the order of days-weeks before most of your user base is up to date. It's easier for say a website or service that can deploy to millions of users within minutes.
- bytematic 7y ago
  Yeah, I agree for sure. I get tons of updates for spotify, discord, and other applications that don't need to do complex binary distribution

therealjumbo 7y ago

This is a great overview of different release management strategies and the practical implications: https://www.youtube.com/watch?v=IKsQsxubuAA https://www.youtube.com/watch?v=IKsQsxubuAA Not part of the talk: if you have more than 3 people involved you probably need a release schedule in terms of either scope or time. I'd personally advise you to have one even if you only have 1 person, just for a mental target if nothing else. The talk goes into studies of projects which did scope based releases and got horrible results in practice. All the ones in the talk switched to time based releases and were much happier. Also, in a software project, you can only fix either scope or time, but not both. Also, if you want to ship in a timely manner, you probably want to fix one of them, not let both float. Edit: it looks like this is his paper on the topic: http://www.cyrius.com/publications/michlmayr_fitzgerald-time_based_release_management.pdf http://www.cyrius.com/publications/michlmayr_fitzgerald-time...

aylmao 7y ago

> Why release on a schedule at all? I feel like it might help with motivation. It's easy to push things "to tomorrow" if you don't have a hard cap on how many tomorrows are available before the feature is supposed to go out. > If people demand a certain new feature ("[this will] bring you new features more quickly"), it can just be brought out as it's ready, instead of having to wait an average of 3 (now 2) weeks before it can be released? It's kind of like forming a habit. Having a set schedule could in theory delay quick features, but it will also keep the motivation going for time-consuming features. > And why does this have to increment the major version, since when is every single update backwards incompatible? I actually really like this personally. Encoding compatibility in a version number is, IMO, useless— it's sometimes just hard to know if your changes will break something or not, so there's a wealth of minor versions out there that break things anyway. Just using a single number just makes everything simpler. Detecting breaking changes should be done with systems (tests, static analysis, etc) and not by trusting a number some developer came up with.

deleted 7y ago

[deleted]

batmansmk 7y ago

For quite a few products, releasing takes time. QA, auditing the security, localizing, generating the builds, executing exhaustive in-depth unit tests, flushing caches, writing release notes, publicating on different websites... it can take a while! When the cost of the release goes above a certain amount, due the sheer number of features or the thoroughness of the release process, it starts to make sense to mutualize release tasks between a bunch of features. Bundling those features on a time basis is quite often a local optimum between an acceptable delay for the product team and an acceptable cost for ops/release/qa/...

wpietri 7y ago

They're clearly moving gradually in this direction, but I think it's an open question how far they can get. For server-side software, it's easy enough to do multiple releases per day, because you have full control over the environment, and all the work is internal. But for desktop/mobile software like this, having multiple releases per day won't be efficient in terms of bandwidth, load, or user experience. And each browser version has other external costs, like documentation and support. Right now it can be hard to solve a problem like, "How do I do X with Firefox" because the answer varies by version. The more versions you have, the harder that is. Or can you imagine trying to triage and verify bug reports if there are, say, 3 new Firefox releases per day, and people could be on any one of dozens of versions because they don't quit and restart the browser very often? We can imagine solutions to these problems, of course. But I think getting a release cadence below weekly for desktop apps will be a very long road.

roca 7y ago

One issue with releasing every day or every hour is that that would mean users downloading massive numbers of updates. Sure, you ship deltas, but there's still a cost to each update.

Thorrez 7y ago

Probably for a similar reason to why c++ releases a new standard every 3 years. This blog post explains c++'s reasoning: https://herbsutter.com/2019/07/13/draft-faq-why-does-the-c-standard-ship-every-three-years/ https://herbsutter.com/2019/07/13/draft-faq-why-does-the-c-s...