10 ms·
How to fuck up software releases
- sroussey 7y agoNice heads up! Working on a similar script to help with updating versions and packaging across five platforms. At least I have a (mostly) monorepo. Which is great until it’s not.
- vortico 7y agoI keep a release TODO list and about 50% of every release, I fuck up and need to add something to the list. It's about 40 items long now (containing various terminal commands and activities), which includes making sure manual entries, documentation, the website, social media, dependencies, etc are handled correctly.
- Swizec 7y agoSounds like a great potential SaaS offering
- z3ugma 7y agoManifest.ly is designed for this workflow - https://www.manifest.ly/ https://www.manifest.ly/
- PacifyFish 7y agoWhat kinds of things do you need to check for that aren’t automated? Maybe I’m spoiled by our CI pipeline. I’d love to take a look at your list
- vortico 7y agoHere's a somewhat redacted version. `m` is my alias for `make`. - Finalize source - Legal check `LICENSE-dist.txt` - Update `CHANGELOG.md` - Bump version in `Makefile` and `Core.json` - Commit "Bump version". - Build (three OS's can be done in parallel) - `m clean` - `git pull` - Make sure you have the latest `Fundamental.zip` package in source root. - `m dist` - Manually test installer and fragile features (audio drivers, patch loading) for ~10 minutes. - `m notarize` (on Mac) - `m upload` - `git tag vX.Y.Z` - `git push --tags` - Release - Update version title and URLs in `Rack.pug` - `m upload` - At this point, normal users have access to new version. - Update server version in `config.coffee` - `m restart` - At this point, normal users will swarm to download new version. Keep an eye on server bandwidth. - Publicize - Twitter https://twitter.com/home - Facebook https://www.facebook.com/vcvrack/ - Share on group - Forum https://community.vcvrack.com/c/announcements
- enriquto 7y agothis list is a very simple algorithm. Why don't you tun it into a script deploy.sh ?
- vortico 7y agoNo it's not.
- enriquto 7y agoBut it almost is, already! Half of the lines are literally shell commands. And except maybe for the legal check, the rest of the steps can also be written as shell one-liners. For example, to update the version on a README file you can "sed -i" with an appropriate replacement, and so on.
- vortico 7y agoCan you explain how your ./deploy.sh script forks onto three laptops (Windows/Mac/Linux) and "plays around" with the installers/build a bit to make sure it works? I obviously can't release builds that don't even run because I didn't take a few minutes to test them.
- enriquto 7y agoWhy don't you use virtual machines?
- vortico 7y agoGood question. 1) A MacOS virtual machine is probably illegal. 2) I need to test the builds on all three machines, and VMs don't handle OpenGL, USB drivers (e.g. for MIDI controllers), and audio drivers very well or at all. 3) I need to have a rough idea of performance of the build. VMs affect performance by non-constant/unpredictable factors. For example, CPU performance might be 1x the speed of a bare-metal OS, but graphics performance might be 0.25x. 4) I don't know how to write a shell script on a host computer that launches Windows 10 and runs commands with an MSYS2 Mingw64 shell.
- gourlaysama 7y agoThe Scala release checklist [1] is a good example of a release checklist with both automated and manual steps. [1]: https://github.com/scala/scala-dev/issues/645 https://github.com/scala/scala-dev/issues/645
- chubot 7y agoEvery time I "fuck up", I add a few commands to a shell script, so I don't mess up the next time. Over time that has grown into over 1000 lines of shell, e.g. https://github.com/oilshell/oil/blob/master/devtools/release.sh https://github.com/oilshell/oil/blob/master/devtools/release... And this is why I'm writing a shell -- because it's good for automating things that you are otherwise too lazy to automate! I find that "semi-automation" is a good middle ground. This is where shell is better than "real" languages. You don't have to commit to writing a polished program -- you just incrementally improve things and it makes problems go away. The time you put in is saved immediately, as opposed to other languages, where you might spent 30 minutes writing something that will save you 2 minutes in the next year, which isn't necessarily worth it. If you keep a text file with a bunch of terminal commands, then you might as well chmod +x it and put #!/bin/sh on the front. That's what a shell script is! ---- Here is the output of the release process, which has tarballs, change logs, documentation, test results, benchmarks, etc. https://www.oilshell.org/release/0.7.pre5/ https://www.oilshell.org/release/0.7.pre5/ It's way to much to do by hand.
- ssully 7y agoIt's so relieving to read this thread. Last year I took over management of some legacy scripts and was tasked with updating them (move from Python2 to 3, interfacing with some new APIs, etc). One of the biggest stresses I had was little fuck ups I kept doing with the releases. It was never anything huge, but every few releases I would end up taking 15-30 minutes longer then I planned, and would involve me updating the release process slightly.
- lonelappde 7y agoFlagged for inflammatory title the pollutes discourse.
- techntoke 7y agoThat is literally the title of the post.
- dang 7y agoSome people find profanity inflammatory in a way that could reasonably be considered linkbait, in which case the site guidelines would require a title change (https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html). There's no solution here that will satisfy everybody. Generally we find that it's better for HN not to edit away profanity. If there's some other reason to edit a title, that's different. But profanity per se is often just energetic and direct language, and if you take it away you run the risk of Bowdlerizing or bringing in euphemisms, which is not in the spirit of this site. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- peterwwillis 7y agoA lot of these can be avoided by making all the steps repeatable. To do that, you should run everything in a Docker container that's built solely from version controlled files, and run it on a remote system that gets blown away regularly. You can use Jenkinsfiles to specify exactly what to do with different branches, and to really simplify things, you can have a different job for every kind of release. Build in a dry-run option so you can run a whole release without actually publishing anything. Each of your software projects can have its own specific release steps in a script run by your release container, while still following the same basic release process for all your projects.
- greggyb 7y agoFor some prior art in this space, it may be useful looking at ESR's shipper: http://www.catb.org/esr/shipper/ http://www.catb.org/esr/shipper/ And some blog posts he's written on it: http://esr.ibiblio.org/?s=shipper&submit=Search http://esr.ibiblio.org/?s=shipper&submit=Search
- kick 7y agoCan dang or sctb fix the title of this to the correct one? "How to fuck up software releases"
- josteink 7y ago> I am doomed to creatively outsmart my tools in releases. That’s a very honest admission, and one I think lots of “why don’t you automate everything”-crowd seems to fail to recognise as a real-world factor.
- praptak 7y agoI would say the article actually supports the "automate everything" position. The author prevents future mistakes by adding more automation rather than less.
- cjblomqvist 7y agoI agree. While I like to automate away things both for saving time and consistent results, we have a tendency to throw away good old thinking or the window. Have I covered everything? Did I check all the outputs? What am I missing? What step that I did last time have I missed this time? Sometimes it feels like we're throwing out good old proper thinking and proper work in order to blame automation, that the results is a natural outcome of all the complexity or that we're pressed for time. Sometimes it's just a matter of taking responsibility and doing a good job.
- perlgeek 7y agoAnother "fun" one: typo in git tag names. The rakudo project has monthly releases with tags like 2019.07. Once I mixed up the year and month in e release, so I created a tag 2018.06 instead of 2016.08 (don't remember the exact numbers), and pushed it. I deleted the wrong tag as soon as I noticed, and pushed the correct one, but of course when the date of the typo'ed release month came around some years later, things blew up for everybody who still had the wrong tag in their local repo (which turned out to be quite some developers).
- m0xte 7y agoThis is one thing I really loved about subversion. I just used the VCS revision as the tag and version therefore avoiding this issue. Didn’t even need to create tags. That was until the version overflowed the 16 bit field it was stored in the assembly manifest. Kaboom. :( People spend way too much time arguing with semantic versioning when 99% of the time the only important thing is the next number is bigger than the previous number.
- jakobegger 7y agoSVN revision numbers are the only thing I miss after switching to git.
- dbrgn 7y agoFor bumping the version there's also bumpversion: https://github.com/peritus/bumpversion https://github.com/peritus/bumpversion
- merlincorey 7y agoTwo things should be noted: 1. `bumpversion` is more general and has more functionality than `semver` in the article 2. `bump2version` is the currently maintained version[0] I currently use `bumpversion` to manage versioning in applications with many different versioning schemes as well as managing the versioning of their deployment environments (such as updating Terraform files, etc). [0] https://github.com/c4urself/bump2version https://github.com/c4urself/bump2version
- lifeisstillgood 7y agoThe big lessons from this article (which I love and resonate with) - You discover more bugs with your shins than your foresight - As long as you keep fixing it in code, you will eventually win - it's like a video game, as long as you can respawn, you will always defeat the game. Just keep buggering on.
- NKosmatos 7y agoI came here expecting to read about how Apple is (mis)treating their macOS lately or how Microsoft has managed to fuck up their windows update process, but instead I find a few good tips while building and numbering software.
- tlocke 7y agoFor Python projects I use the excellent Versioneer: https://github.com/warner/python-versioneer https://github.com/warner/python-versioneer It means you only specify the version as a git tag, and all the other things that need a version number get it from that.
- kissgyorgy 7y agoA thousand ways. One really good idea I read was that you write an empty script, which prints the instructions for each step and you press enter, it prints the next step. It can only print instructions at first, then later, you can write code to automate some (maybe even validate things) steps. Something like this: def git_tag(): print("1. git tag the repository") input("Press enter to continue") def git_push(): print("2. git push the repository") input("Press enter to continue") def pip_deploy(): print("3. Deploy, run: python setup.py sdist upload") input("Press enter to continue") def main(): git_tag() git_push() pip_deploy() As simple as that! Then later, you could write code for every step as you have time.
- x1798DE 7y ago> print("3. Deploy, run: python setup.py sdist upload") The setup.py upload command is actually deprecated and has been replaced with "twine upload". https://setuptools.readthedocs.io/en/latest/setuptools.html#upload-upload-source-and-or-egg-distributions-to-pypi https://setuptools.readthedocs.io/en/latest/setuptools.html#... https://pypi.org/project/twine/ https://pypi.org/project/twine/
- silvester23 7y agoI think this might be the article you read: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...
- kissgyorgy 7y agoYes! Thanks! I couldn't find it.
- blesi 7y agoAt Braintree, we developed a framework for this: https://github.com/braintree/runbook https://github.com/braintree/runbook One of our biggest use cases is for including ample sanity checks as part of our releases to make sure we don't mess it up.
- yitchelle 7y agoDon't do it on Friday. On a serious note, I have found that the only way to not fuck up is to remove as much human interaction from the release process as possible, which means scripting. Even with scripting, if the inputs are bad, the output will also be bad.
- mattacular 7y agoThat's the point of this post. Automation is hard and probably won't ever be perfect. You can't remove human interaction from software development.
- thrower123 7y agoSeriously, though, don't release on Friday. No matter how good your CI/CD pipeline is, and how confident you are in your testing, you'll let some bad releases through, and if you release on Friday, some poor bastard, whether you or your coworker, is going to be left holding the bag, and have their weekend ruined scrambling around to fix what is broken. It's just inconsiderate. Things can wait until Monday.
- Intermernet 7y agoI'm waiting for the next "disruptor" to make it a sales pitch: "Move fast, break things, and only release on Friday!"
- pnako 7y agoIn fact it's standard practice in the financial industry to deploy everything on the week-end, and indeed starting from Friday night, because that's when most markets are closed. Obviously don't do that if you don't have to, but sometimes it's actually the best time to deploy stuff.
- caseymarquis 7y agoAutomating builds is fun! On the one hand I'd be worried if I joined a team or contributed to a project which still had manual processes for release more complex than 'run this script' or 'push to the x branch'. On the other hand, that's a fun initial project. When developers aren't (or are prevented from) improving their own environment, that's a bad sign...
- mcnichol 7y agoI love the humble nature of this post and how fallible we are. I have spent many days playing whack-a-mole with issues and hacking out solutions. What is equally great is the community sharing their gotchas and how they solved them. A wholesome and organic thread, thanks for taking the time to write it!
- mugsie 7y agoI really like how the OpenStack project does releases - it has a ton of tooling, but for people who are deciding what to release, it is super simple. e.g. - the latest RC of a project I maintain is L10-14 on https://opendev.org/openstack/releases/src/branch/master/deliverables/train/designate.yaml https://opendev.org/openstack/releases/src/branch/master/del... by committing that file, the tooling chooses the right commit, does a tag, and builds the python tarballs, and puts them on releases.openstack.org, and for the client libraries, it pushes them to pypi. it avoids a lot of the footguns of special scripts in repos, while allowing an easy release process.
- thedancollins 7y agoPhft, amateur. I've forgotten more ways to screw stuff up than this person ever knew.
- RexM 7y agoThis reads like the "Deploying on a Friday Afternoon Greatest Hits".
- aepiepaey 7y ago> Update the signing script to save the tarball to disk (previously, it lived in a pipe) and upload these alongside the releases… This should have been done from the start. GitHub does not guarantee checksums for the generated source archives to be stable, so they can change when GitHub updates their software (and yes, this has happened).
- jillesvangurp 7y agoA couple of practical solutions here. 1) Automate your releases and run that on a CI server if you can. "It works on my laptop" is not ideal for releasing stuff. 2) Have a release branch, and only allow stuff to be merged that passes your CI tests and trigger the automation under step 1 after a merge to your release branch. This works for continuous deployment but can also work for releases of other stuff. If you use semantic versioning, the CI server should tag the release and publish the artifacts. That's one less thing that can go wrong. If you forgot to bump the version number on your master branch, the build should fail. 3) Have a release checklist. People forget stuff and having a small list of "Are the release notes and readme updated? Is our CI not complaining? Have all PRs been merged? Etc. Even with the above, premature releases through an early merge to your release branch could happen. A checklist can prevent that. And of course, if you can integrate those checklists into your CI. If your release notes are unchanged since the last version 4) Release often. Small deltas are less work to test and less risky to impose on users and you get feedback on stuff you did earlier.