9 ms·

How to fuck up software releases (2019)

119 points by WayToDoor 5y ago

nickcw 5y ago
I loved this article - it is so my life. > I manage releases for a bunch of free & open-source software. Just about every time I ship a release, I find a novel way to fuck it up. With rclone I have got nearly everything scripted now after 84 releases over 8 years, but I regularly mess up on other repos. Releasing a feature branch is something I've done more than once, then merging the feature branch (with rebase) and wondering where on earth the previous version tag is - that is a particular favourite. The main thing I've learnt from all these mistakes is to write a RELEASE.md in every repo with instructions on how to do the release. You might only do a release every 6 months and you will have forgotten how to do it the next time. Update the RELEASE.md and the release tools every time you mess up and at least you'll be learning from your mistakes!
- rand49an 5y ago
  You're clearly doing something right, Rclone is a fantastic tool!
- samus 5y ago
  Something similar happened to me recently: I updated all the pom.xml files to the correct version and then created the corresponding tag... in the wrong repository. Yes, wouldn't have happened with Git.
- kodah 5y ago
  A release.md is good for eliminating the bus factor, but it does not prevent errors.
  - SamWhited 5y ago
    It doesn't prevent all errors, but having a script to follow definitely will stop some errors.
  - drewzero1 5y ago
    Prevent errors, no. Reduce the likelihood of making the same error multiple times out of forgetfulness? Probably.
- wildmanx 5y ago
  The way out is to never do anything by hand. Always put it in your release script and run the script from an as-dirty-as-possible workspace. Whenever you do something manually, this manual process that exists as a kind of "virtual script" in your head gets out of sync with the script that you have on disk. The article describes the continuous struggle to keep those two in sync. By only having the one on disk, that struggle goes away.
taleodor 5y ago
Part of the functionality we built in Reliza Hub is to generate automatic version increments that are guaranteed to be unique on each request and have special rules for versioning on feature and release branches. My usual versioning convention for a feature branch is BRANCH_NAME.PATCH Generally, I believe getting versioning right requires an external versioning service that ensures uniqueness and routing between branches. Otherwise there will always be "gotchas".
zild3d 5y ago
I'd think making it a CI step is more reliable than a manual script with all the mentioned footguns. e.g. a Github action can be configured to only run on the desired branch, with specific tags, etc. Doesn't matter if the commit was from Bob or Billy's computer
- ddevault 5y ago
  A CI is made out of code and configuration files and policies which can also be screwed up. Using CI can be helpful but it won't save you from these kinds of mistakes.
  - bo0tzz 5y ago
    It narrows it down though. The release process is a bunch of steps, and errors are either in the process spec or in the execution of it (human error). Using a CI you remove the possibility of human error, leaving only the option of spec errors - that seems like an improvement to me.
  - theptip 5y ago
    This is true, but you get a good separation of concerns by having your CI/CD server be the thing that strictly decides what SHA/tag/branch triggers the job, and your job is written to be flexible enough to build/deploy an arbitrary commit as instructed by the build server or the developer. For example, I've built Gitlab pipelines with a special job for tags to do the release/deploy, and a manually-triggered job that only runs on master to `bumpversion minor; git push tag $NEWVERSION` for minting the release candidate commit -- this way you push a "bump minor version" button in your CI server, and that job by construction only runs on master, is always the latest master, must be a clean repo, etc. With this pattern you're still free to manually run the job if you need to do a patch release in a hurry, but the happy path is constrained. (Also, I recommend to run your build/deploy scripts inside a container, that way your local build env is by construction the same as your build server's. Then you don't have the "different build output when I run from a different flavor of Linux/UNIX" problem.)
  - eska 5y ago
    You can screw the automation up, but when you fix the error it's gone forever. Compare this to having to do 5 manual steps every single time, potentially under stress. At some point you'll mess up. The errors in CI instead are systematic (like deciding on the wrong 5 manual steps).
aequitas 5y ago
It's too bad this video is in German and a little outdated, but it does a great job at introducing the virtues of CI/CD using a movie (Indiana Jones) which not many people would associate with software deployment: https://www.youtube.com/watch?v=eF-RJBEdMOQ&t=93s https://www.youtube.com/watch?v=eF-RJBEdMOQ&t=93s
ncmncm 5y ago
I am forced to wonder if releasing code while on vacation is not itself an overwhelmingly more serious fuckup than anything that happened in it. Then, doing it again.
- motleymage 5y ago
  There is a reason you don’t release on Friday, but I agree, releasing while on a vacation may actually be worse
Xunjin 5y ago
Ty for this, helps me at my "Impostor" to actually see that people which "are in whole other level", do the same mistakes that I've done :P
Mathnerd314 5y ago
> To address these issues, I added a custom .git/_incr_version script which can add additional logic on a per-repo basis, and updated semver to call this script if present. The article mentions several problems caused by this. An alternative method would be to check the semver script into each project repository. This is the way things tend to be done with CMake modules. It does make updating all usages of the script harder, but these sorts of scripts change once every few years if that.
- samus 5y ago
  I have a similar situation at work. What about making this script more generic, factoring it out into its own repo and referencing it with Git submodules or SVN externals? You could also think about turning it into a proper plugin for the package manager.
  - Mathnerd314 5y ago
    To paraphrase Jamie Zawinski: Some people, when confronted with a problem, think "I know, I'll use Git submodules." Now they have two problems. The real issue is finding a home. Someone's personal dotfiles aren't really usable, maybe https://github.com/tj/git-extras https://github.com/tj/git-extras or some project along those lines.
client4 5y ago
I highly value articles like this one -- open and honest; it peels back the beautiful veneer to show great work has obstacles and perfection is always a struggle.
ChrisArchitect 5y ago
More previous discussion: https://news.ycombinator.com/item?id=21244352 https://news.ycombinator.com/item?id=21244352
gjjvdburg 5y ago
I've been using a "do-nothing script" [1] to semi-automate the release process for some open source projects (e.g. [2]). Might be interesting for someone here. [1]: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-... [2]: https://github.com/alan-turing-institute/CleverCSV/blob/master/make_release.py https://github.com/alan-turing-institute/CleverCSV/blob/mast...
devoutsalsa 5y ago
TIL I forgot to create/update the CHANGELOG on a couple projects version I pushed not too long ago :)
meristem 5y ago
General comment: for everyone who tracks and attempts to not repeat whatever just botched up your release: at least the new errors are not the same as the old errors!
bocklund 5y ago
A project I work on has a `RELEASING.rst` in the repo, which I include in the docs [1]. It contains a checklist of steps with the code. The actual deployment is automated with a GitHub Action to deploy to PyPI. The checklist makes sure I hit all the things I need to do. There's still places to screw up the manual steps, replacing the right versions, etc. It's not perfect and there's a bunch of things I want to improve still. I try to write it clearly enough so that someone who's never released a project ever just needs to follow the steps. [1] https://espei.org/en/latest/releasing.html https://espei.org/en/latest/releasing.html

119 points by WayToDoor 5y ago

I loved this article - it is so my life. > I manage releases for a bunch of free & open-source software. Just about every time I ship a release, I find a novel way to fuck it up. With rclone I have got nearly everything scripted now after 84 releases over 8 years, but I regularly mess up on other repos. Releasing a feature branch is something I've done more than once, then merging the feature branch (with rebase) and wondering where on earth the previous version tag is - that is a particular favourite. The main thing I've learnt from all these mistakes is to write a RELEASE.md in every repo with instructions on how to do the release. You might only do a release every 6 months and you will have forgotten how to do it the next time. Update the RELEASE.md and the release tools every time you mess up and at least you'll be learning from your mistakes!

rand49an 5y ago
You're clearly doing something right, Rclone is a fantastic tool!
samus 5y ago
Something similar happened to me recently: I updated all the pom.xml files to the correct version and then created the corresponding tag... in the wrong repository. Yes, wouldn't have happened with Git.
kodah 5y ago
A release.md is good for eliminating the bus factor, but it does not prevent errors.
- SamWhited 5y ago
  It doesn't prevent all errors, but having a script to follow definitely will stop some errors.
- drewzero1 5y ago
  Prevent errors, no. Reduce the likelihood of making the same error multiple times out of forgetfulness? Probably.
wildmanx 5y ago
The way out is to never do anything by hand. Always put it in your release script and run the script from an as-dirty-as-possible workspace. Whenever you do something manually, this manual process that exists as a kind of "virtual script" in your head gets out of sync with the script that you have on disk. The article describes the continuous struggle to keep those two in sync. By only having the one on disk, that struggle goes away.

taleodor 5y ago

Part of the functionality we built in Reliza Hub is to generate automatic version increments that are guaranteed to be unique on each request and have special rules for versioning on feature and release branches. My usual versioning convention for a feature branch is BRANCH_NAME.PATCH Generally, I believe getting versioning right requires an external versioning service that ensures uniqueness and routing between branches. Otherwise there will always be "gotchas".

zild3d 5y ago

I'd think making it a CI step is more reliable than a manual script with all the mentioned footguns. e.g. a Github action can be configured to only run on the desired branch, with specific tags, etc. Doesn't matter if the commit was from Bob or Billy's computer

ddevault 5y ago
A CI is made out of code and configuration files and policies which can also be screwed up. Using CI can be helpful but it won't save you from these kinds of mistakes.
- bo0tzz 5y ago
  It narrows it down though. The release process is a bunch of steps, and errors are either in the process spec or in the execution of it (human error). Using a CI you remove the possibility of human error, leaving only the option of spec errors - that seems like an improvement to me.
- theptip 5y ago
  This is true, but you get a good separation of concerns by having your CI/CD server be the thing that strictly decides what SHA/tag/branch triggers the job, and your job is written to be flexible enough to build/deploy an arbitrary commit as instructed by the build server or the developer. For example, I've built Gitlab pipelines with a special job for tags to do the release/deploy, and a manually-triggered job that only runs on master to `bumpversion minor; git push tag $NEWVERSION` for minting the release candidate commit -- this way you push a "bump minor version" button in your CI server, and that job by construction only runs on master, is always the latest master, must be a clean repo, etc. With this pattern you're still free to manually run the job if you need to do a patch release in a hurry, but the happy path is constrained. (Also, I recommend to run your build/deploy scripts inside a container, that way your local build env is by construction the same as your build server's. Then you don't have the "different build output when I run from a different flavor of Linux/UNIX" problem.)
- eska 5y ago
  You can screw the automation up, but when you fix the error it's gone forever. Compare this to having to do 5 manual steps every single time, potentially under stress. At some point you'll mess up. The errors in CI instead are systematic (like deciding on the wrong 5 manual steps).

aequitas 5y ago

It's too bad this video is in German and a little outdated, but it does a great job at introducing the virtues of CI/CD using a movie (Indiana Jones) which not many people would associate with software deployment: https://www.youtube.com/watch?v=eF-RJBEdMOQ&t=93s https://www.youtube.com/watch?v=eF-RJBEdMOQ&t=93s

ncmncm 5y ago

I am forced to wonder if releasing code while on vacation is not itself an overwhelmingly more serious fuckup than anything that happened in it. Then, doing it again.

motleymage 5y ago
There is a reason you don’t release on Friday, but I agree, releasing while on a vacation may actually be worse

Xunjin 5y ago

Ty for this, helps me at my "Impostor" to actually see that people which "are in whole other level", do the same mistakes that I've done :P

Mathnerd314 5y ago

> To address these issues, I added a custom .git/_incr_version script which can add additional logic on a per-repo basis, and updated semver to call this script if present. The article mentions several problems caused by this. An alternative method would be to check the semver script into each project repository. This is the way things tend to be done with CMake modules. It does make updating all usages of the script harder, but these sorts of scripts change once every few years if that.

samus 5y ago
I have a similar situation at work. What about making this script more generic, factoring it out into its own repo and referencing it with Git submodules or SVN externals? You could also think about turning it into a proper plugin for the package manager.
- Mathnerd314 5y ago
  To paraphrase Jamie Zawinski: Some people, when confronted with a problem, think "I know, I'll use Git submodules." Now they have two problems. The real issue is finding a home. Someone's personal dotfiles aren't really usable, maybe https://github.com/tj/git-extras https://github.com/tj/git-extras or some project along those lines.

client4 5y ago

I highly value articles like this one -- open and honest; it peels back the beautiful veneer to show great work has obstacles and perfection is always a struggle.

ChrisArchitect 5y ago

More previous discussion: https://news.ycombinator.com/item?id=21244352 https://news.ycombinator.com/item?id=21244352

gjjvdburg 5y ago

I've been using a "do-nothing script" [1] to semi-automate the release process for some open source projects (e.g. [2]). Might be interesting for someone here. [1]: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-... [2]: https://github.com/alan-turing-institute/CleverCSV/blob/master/make_release.py https://github.com/alan-turing-institute/CleverCSV/blob/mast...

devoutsalsa 5y ago

TIL I forgot to create/update the CHANGELOG on a couple projects version I pushed not too long ago :)

meristem 5y ago

General comment: for everyone who tracks and attempts to not repeat whatever just botched up your release: at least the new errors are not the same as the old errors!

bocklund 5y ago

A project I work on has a `RELEASING.rst` in the repo, which I include in the docs [1]. It contains a checklist of steps with the code. The actual deployment is automated with a GitHub Action to deploy to PyPI. The checklist makes sure I hit all the things I need to do. There's still places to screw up the manual steps, replacing the right versions, etc. It's not perfect and there's a bunch of things I want to improve still. I try to write it clearly enough so that someone who's never released a project ever just needs to follow the steps. [1] https://espei.org/en/latest/releasing.html https://espei.org/en/latest/releasing.html