6 ms·

The post tangentially touches on a pet peeve of mine that I think lots of companies get fundamentally wrong these days: version control, CI, and deployment are

by pdpi 5y ago

The post tangentially touches on a pet peeve of mine that I think lots of companies get fundamentally wrong these days: version control, CI, and deployment are separate concerns and shouldn't be jumbled together into an amorphous blob. If the build fails, you should be able to re-run it without pushing to version control. You can e.g. have builds fail due to issues with your artifact repository that don't require any changes to the code to debug/fix. You can have issues with deployment after the build completes, you don't want to have to rebuild to redeploy. The most shocking of all, though, and I have actually seen this: If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control. As always, it's very much a case of too much of a good idea. People correctly identify that automation is a good thing, but then turn that into "manual operation is a bad thing, and therefore we shouldn't cater to it".

rileymat2 5y ago
> If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control. Why not? In some cases in lighter weight systems, you might have a release branch that gets pushed to production. Like dokku.
- pdpi 5y ago
  Key word there is "require". Having that automation set up is super convenient and a pretty important factor in making things more productive, not contesting that. The other side of the equation is that if I break something, I want to make the smallest change possible that fixes it, so as to avoid confounding variables if it doesn't work. Off the top of my head: * I don't want to make things worse by messing up the rollback due to stress * I don't want to trigger a build and wait for CI to run and build * I don't want to depend on my build not being perfectly reproducible (e.g. because the problem comes from a bad dependency upgrade and the rollback rebuilds with the bad dependency) What I do want is a really simple way to just tell my system "You know that thing you were running 5 minutes ago? Please go back to running that again".
llamataboot 5y ago
Don't all action runners allow you to re-run tests and builds with a click of a button and most platforms allow rolling back to any arbitrary commit number? I have never seen any of the above issues.
- llamataboot 5y ago
  (also I don't agree on not rolling back, I want one branch to always be the actual code living on prod, or else you end up with arbitrary time wasting rabbitholes of accidentally debugging code that isn't live)
  - jyounker 5y ago
    So you're opposed to practice of canary deployments?
smirgel 5y ago
Thank you! I thought I was alone in thinking like this. I hate Atlassian Bamboo with a passion but it is the only CI/CD system I remember coming across that actually has a concept of deploying a built artifact.
- solarengineer 5y ago
  Have you tried GoCD?
  - smirgel 5y ago
    I haven't yet. Good to know there are options. Thanks!
deleted 5y ago
[deleted]
015a 5y ago
I kind of disagree. While obviously these three things (VCS, CI, and Deployments) are different things, there's a lot of value in reducing all of them to just one thing: VCS is truth. It can make the system far easier to reason about. Question 1: What's running in production? Allowing production to run "anything" means "anything" could be running there, so you need tooling to answer that question. Maybe that's just "give every developer read-level access to AWS", or maybe its a slack slash command, or something, but there's a cost there. With a coordinated model: its just the main branch. Its always the main branch. So look at the main branch. (Sidenote: this isn't a perfect solution, because oftentimes there's a temporal delay between a main branch merge and deployment, and developers want to know precisely when the cutover starts/finishes. So you end up building that extra stuff anyway. But: that can be delayed quite a bit, as its a higher level question given that the main-is-production model solves a lot of problems alone). Question 2: How do I re-run the build? Well, you can push a new commit, or issue some slack slash command, or maybe a comment in the PR that reads `!buildbot run` or whatever. How about we just make it simpler? Just push a new commit. The broader point is that: turing machines can do anything. Human brains can't. So when operations are simplified around models like "source-is-truth" its because that model is a satisfying fit for the problem space which helps the human brain reason about what is happening. All models are wrong; but some are useful.
- ttmb 5y ago
  > VCS is truth VCS can only ever be intent at best.
  - raffraffraff 5y ago
    I agree. This goes back to the recent post "don't use latest". If you don't pin everything you can't claim that VCS is truth. A built binary is the truth, whether that be a container image, go bin or whatever. Even then I wouldn't say it's the absolute truth. The whole system is the truth.
  - 015a 5y ago
    "VCS is truth" does not assert what it is, but rather its treatment in the system. There is no knowable truth in large-scale systems design. Velocity is relative in physics, right? Relative to what? What's the frame of reference? Usually, we say, the Earth itself. VCS is that to software engineering, in this setup; there's movement and realities outside of VCS, but VCS is our zero; the tare. Everything is relative to the VCS. It is not truth through its nature; it is truth through how we treat it, how we build systems to interface with it, synchronize with it, etc. All models are wrong, but some are useful.
deathanatos 5y ago
> If the build fails, you should be able to re-run it without pushing to version control. I've uh, not seen one that doesn't? (I agree with you, it shouldn't. Does your CI system not have a "retry" or "restart" button?) > The most shocking of all, though, and I have actually seen this: If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control. So, my company does this, and I love it; it gives even the least skilled of our devs a good shot at the "recognize things aren't working → problem started with last deploy → revert last deploy" debug path, and getting it right. Having that last step of revert a deploy literally correspond to a git revert is very useful & easy. Keeping the "state of what is deployed" in code also gives us good confidence that there aren't random changes being made, allows for code review, provides a log of who changed what when — all the goodness of infrastructure as code. Now, we keep the main production "monorepo" separate from our repo that describes what is deployed (let's call it "deploy-state". So a revert of a deployment happens in the "deploy-state" repo, not in the monorepo. So, someone's merged feature branch isn't going to get reverted. (Though, we should have a conversation about whatever the actual problem is: if it is code in the monorepo, why did it not fail tests there? We also have ways of saying "this range of commits, from [break, fix), should not get deployed to prevent repeats.) Whether that setup is a "true monorepo" or not, since it literally involves more than one repo … well, you be the judge. It seemed to us pragmatic to have it be separate, and we've not regretted that.
- diablerouge 5y ago
  This isn't that very different from having a "deploy-state" branch that runs alongside your master branch too, bringing it even closer to a true monorepo.
- pdpi 5y ago
  > So a revert of a deployment happens in the "deploy-state" repo, not in the monorepo. This makes me think you're disagreeing with the literal interpretation of what I wrote, while very much agreeing in spirit. I'm not trying to say that a VCS is an intrinsically wrong solution to managing deployment state (I'm not a fan, but it's an eminently reasonable approach). My issue is with deployment state and application source being enmeshed in the same repo such that you can't evolve one without messing with the other.
HelloNurse 5y ago
Another class of problem, even without errors, is needlessly restricted concurrent access: your teams tries to push an urgent fix that needs to be built, tested and deployed an hour ago, to find that the CI system will be on its knees, or actually locked, because another team is compiling and testing something unimportant. In many cases, insult can be added to injury by forcing you to waste more time after the other build finishes to change version numbers or merge completely irrelevant changes in source control.
- withinboredom 5y ago
  We have reverts bypass testing and most of CI. A revert commit on master goes directly to production as fast as possible. Nothing should stop an emergency revert. If it’s not an emergency, it goes through the normal process.

by pdpi 5y ago

rileymat2 5y ago

> If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control. Why not? In some cases in lighter weight systems, you might have a release branch that gets pushed to production. Like dokku.

pdpi 5y ago
Key word there is "require". Having that automation set up is super convenient and a pretty important factor in making things more productive, not contesting that. The other side of the equation is that if I break something, I want to make the smallest change possible that fixes it, so as to avoid confounding variables if it doesn't work. Off the top of my head: * I don't want to make things worse by messing up the rollback due to stress * I don't want to trigger a build and wait for CI to run and build * I don't want to depend on my build not being perfectly reproducible (e.g. because the problem comes from a bad dependency upgrade and the rollback rebuilds with the bad dependency) What I do want is a really simple way to just tell my system "You know that thing you were running 5 minutes ago? Please go back to running that again".

llamataboot 5y ago

Don't all action runners allow you to re-run tests and builds with a click of a button and most platforms allow rolling back to any arbitrary commit number? I have never seen any of the above issues.

llamataboot 5y ago
(also I don't agree on not rolling back, I want one branch to always be the actual code living on prod, or else you end up with arbitrary time wasting rabbitholes of accidentally debugging code that isn't live)
- jyounker 5y ago
  So you're opposed to practice of canary deployments?

smirgel 5y ago

Thank you! I thought I was alone in thinking like this. I hate Atlassian Bamboo with a passion but it is the only CI/CD system I remember coming across that actually has a concept of deploying a built artifact.

solarengineer 5y ago
Have you tried GoCD?
- smirgel 5y ago
  I haven't yet. Good to know there are options. Thanks!

deleted 5y ago

[deleted]

015a 5y ago

I kind of disagree. While obviously these three things (VCS, CI, and Deployments) are different things, there's a lot of value in reducing all of them to just one thing: VCS is truth. It can make the system far easier to reason about. Question 1: What's running in production? Allowing production to run "anything" means "anything" could be running there, so you need tooling to answer that question. Maybe that's just "give every developer read-level access to AWS", or maybe its a slack slash command, or something, but there's a cost there. With a coordinated model: its just the main branch. Its always the main branch. So look at the main branch. (Sidenote: this isn't a perfect solution, because oftentimes there's a temporal delay between a main branch merge and deployment, and developers want to know precisely when the cutover starts/finishes. So you end up building that extra stuff anyway. But: that can be delayed quite a bit, as its a higher level question given that the main-is-production model solves a lot of problems alone). Question 2: How do I re-run the build? Well, you can push a new commit, or issue some slack slash command, or maybe a comment in the PR that reads `!buildbot run` or whatever. How about we just make it simpler? Just push a new commit. The broader point is that: turing machines can do anything. Human brains can't. So when operations are simplified around models like "source-is-truth" its because that model is a satisfying fit for the problem space which helps the human brain reason about what is happening. All models are wrong; but some are useful.

ttmb 5y ago
> VCS is truth VCS can only ever be intent at best.
- raffraffraff 5y ago
  I agree. This goes back to the recent post "don't use latest". If you don't pin everything you can't claim that VCS is truth. A built binary is the truth, whether that be a container image, go bin or whatever. Even then I wouldn't say it's the absolute truth. The whole system is the truth.
- 015a 5y ago
  "VCS is truth" does not assert what it is, but rather its treatment in the system. There is no knowable truth in large-scale systems design. Velocity is relative in physics, right? Relative to what? What's the frame of reference? Usually, we say, the Earth itself. VCS is that to software engineering, in this setup; there's movement and realities outside of VCS, but VCS is our zero; the tare. Everything is relative to the VCS. It is not truth through its nature; it is truth through how we treat it, how we build systems to interface with it, synchronize with it, etc. All models are wrong, but some are useful.

deathanatos 5y ago

> If the build fails, you should be able to re-run it without pushing to version control. I've uh, not seen one that doesn't? (I agree with you, it shouldn't. Does your CI system not have a "retry" or "restart" button?) > The most shocking of all, though, and I have actually seen this: If you deploy a broken build, a rollback to an earlier version should never, ever, require a rollback on your source control. So, my company does this, and I love it; it gives even the least skilled of our devs a good shot at the "recognize things aren't working → problem started with last deploy → revert last deploy" debug path, and getting it right. Having that last step of revert a deploy literally correspond to a git revert is very useful & easy. Keeping the "state of what is deployed" in code also gives us good confidence that there aren't random changes being made, allows for code review, provides a log of who changed what when — all the goodness of infrastructure as code. Now, we keep the main production "monorepo" separate from our repo that describes what is deployed (let's call it "deploy-state". So a revert of a deployment happens in the "deploy-state" repo, not in the monorepo. So, someone's merged feature branch isn't going to get reverted. (Though, we should have a conversation about whatever the actual problem is: if it is code in the monorepo, why did it not fail tests there? We also have ways of saying "this range of commits, from [break, fix), should not get deployed to prevent repeats.) Whether that setup is a "true monorepo" or not, since it literally involves more than one repo … well, you be the judge. It seemed to us pragmatic to have it be separate, and we've not regretted that.

diablerouge 5y ago
This isn't that very different from having a "deploy-state" branch that runs alongside your master branch too, bringing it even closer to a true monorepo.
pdpi 5y ago
> So a revert of a deployment happens in the "deploy-state" repo, not in the monorepo. This makes me think you're disagreeing with the literal interpretation of what I wrote, while very much agreeing in spirit. I'm not trying to say that a VCS is an intrinsically wrong solution to managing deployment state (I'm not a fan, but it's an eminently reasonable approach). My issue is with deployment state and application source being enmeshed in the same repo such that you can't evolve one without messing with the other.

HelloNurse 5y ago

Another class of problem, even without errors, is needlessly restricted concurrent access: your teams tries to push an urgent fix that needs to be built, tested and deployed an hour ago, to find that the CI system will be on its knees, or actually locked, because another team is compiling and testing something unimportant. In many cases, insult can be added to injury by forcing you to waste more time after the other build finishes to change version numbers or merge completely irrelevant changes in source control.

withinboredom 5y ago
We have reverts bypass testing and most of CI. A revert commit on master goes directly to production as fast as possible. Nothing should stop an emergency revert. If it’s not an emergency, it goes through the normal process.