6 ms·

> It’s like DevOps and CI/CD: more frequent deploys are safer. This happens because each deploy contains fewer changes. Is this generally true? Deploy as in go

> It’s like DevOps and CI/CD: more frequent deploys are safer. This happens because each deploy contains fewer changes. Is this generally true? Deploy as in going to production? I join a web app project that didn't have any tests and the code was brittle, where changing one part of the app could break a completely different part without you realising. It was much more efficient (and less stressful) to batch up lots of small changes and do one thorough manual testing session of the whole thing while we gradually got the codebase under control. It likely depends on how much automated testing you have, how bad it is if bugs go live and how quickly you need features live.

jagged-chisel 6y ago
It’s true that deploying frequently reduce the surface of potential new problems per release. But to make it work flawlessly (not perfectly, but approaching) you do need some engineering rigor in the product.
lostcolony 6y ago
It's generally true, yes. While manual testing changes the equation some (the testing effort is the same for each release; the more you can automate that, the greater gains), how easy to -fix- the issue becomes more complicated the more changes there are. 1 PR, you know what introduced the breakage. 30 far reaching ones, with interesting overlap between them? Good luck.
- seanwilson 6y ago
  > 1 PR, you know what introduced the breakage. 30? Good luck. To be clearer, I mean there would be say 30 pull requests merged into "develop" and they would only be merged into "master" and pushed live after detailed manual testing. This is versus merging each pull request directly into "master" and going live immediately, where you wouldn't have time to do detailed manual testing every merge. You would still have the git history in both approaches to debug.
  - lostcolony 6y ago
    Right. I'm saying there are two different things here. Regardless of your merge strategy. You either test after each PR merge, or you test only when you 'craft a release'. Doesn't matter whether this merge is into a develop branch or straight into master; it's solely a question of when you test it. If you manually test, testing only a batch of changes, together, is obviously easier, from a testing perspective. However, the effort to figure out and fix the issue can be very large. Plus, after fixing that one issue, you have to retest everything. So every bug requires retesting in any case (so while it's still a lower testing burden, unless you introduced 30+ unrelated issues, it's not as low as you think it might be on the surface). Compared with testing (and at that point you might as well release if your pipeline and org let you) with each PR - any manual testing effort is obviously higher due to testing so often (whereas automated is no extra effort), but, figuring out what caused the breakage, and addressing it, is much, much easier, order of magnitude easier, to figure out, and fixing it is much less likely to introduce new issues. Which is more to the original point; catching a bug earlier, allows the fix to be more targeted, which makes it more likely to be right, which reduces the likelihood of it making it to prod. There is also, to the original point, a statistical thing to consider. Let's say in 30 PRs, there is one that introduces a bug. Well, if QA is testing the one PR that broke something, they're more likely to spot it, since they're focusing on that PR. They're less likely if it's one of thirty sets of changes. But beyond that, let's say, in both situations, QA misses it. If you have one deploy, you've broken prod 100% of your releases (1 of 1 release). If you have 30 deploys, you've broken prod only ~3% of your releases. So immediately the original statement is validated; if your testing effort is the same (big if!), more frequent releases, with smaller changes, means the same or better percentage of deploying working code. Also, if you have to rollback with one mega release, you lose all 29 other changes; if you have to rollback with separate changes, you lose no other changes.
JamesBarney 6y ago
It's true in the sense that # of bugs or downtime per release is smaller. But that's not what clients care about at all. They care about total number of bugs and downtime. In addition this completely ignores that if you spend the same amount of time testing your testing per release drops when you increase the number of deployments.
- johnmaguire2013 6y ago
  > In addition this completely ignores that if you spend the same amount of time testing your testing per release drops when you increase the number of deployments. No, you just automate your testing.
  - JamesBarney 6y ago
    I've never worked anywhere where 100% of testing was automated.
    - johnmaguire2013 6y ago
      In practice, automating around 90% of it is enough. This does explain why it didn't work for you though.
cortesoft 6y ago
If your code is that brittle, than sure.
shepherdjerred 6y ago
I would hate to work somewhere that my changes are deployed within days of writing my code. CI/CD lets you iterate faster and it encourages best practices like proper testing. That said, if you do CI/CD without proper testing then you're going to break production.
- seanwilson 6y ago
  Wait until you have to work with clients that don't even have staging environments let alone automated testing. Source control, CI/CD, automated testing, pull requests, code reviews, feature flags, linters, good logging etc. might be the norm on HN but not everywhere, especially companies where tech isn't the focus. It can also be a tough sell to make process improvements like this when the old way got them this far and they want new features now. I've worked with clients that don't even use source control, do all edits on production and there's no way setup to develop locally.
  - pletnes 6y ago
    Although my current day job is very professional about this, I’ve seen the other side, too. «We have some code from 5-10 y back, can you compile it?» and only half the files are there. It would be great if HN had some content on «moving from random files on a network drive to git and basic automated testing» - it’s not an easy job and others would benefit from such writeups.
    - oblio 6y ago
      Joel on Software: go through the "Rock Star Developer" section of his archive.
- lmm 6y ago
  > I would hate to work somewhere that my changes are deployed within days of writing my code. Did you miss a "not" here? For me an environment where my changes are deployed and used quickly is a lot more fulfilling than one where I work on something that may only see the light of day weeks or months down the line.
lalaithion 6y ago
If you have decent automated testing, canarying, and monitoring, then yeah, releasing as frequently as possible means that usually a breakage is immediately narrowed down to 2-3 CLs. We release once every 2 hours, because it takes about 2 hours to cut, test, and slowly deploy one release. When we have a bug, we usually notice during the canarying process, and then roll back and fix it, and figure out how to test for that case.
virgilp 6y ago
That's the whole idea of CD: if you set up your project from the start so that deploys go to production, it changes the team culture and the way of working. People don't commit code without any tests (because it will break production!) and conversely don't stress over manually testing dozens of scenarios, some of which were "one-offs" that may become forgotten - it's "normal" that your code gets released, so the release is no longer a large, time-consuming and stressful event. If you start with a project where code is brittle and there are no tests, it is indeed extremely hard and time consuming to move to CI/CD. Doesn't mean it's not a good idea, just that it won't be easy and it'll take time (you have to build the whole infrastructure of automated testing, canary deployments, etc. - but, it's a good idea to do that anyway! It'll gradually improve the velocity. The alternative is that you end up in a world where everybody is afraid of making changes and the simplest requirement ends up getting estimates like "3 months of work".
- js8 6y ago
  > People don't commit code without any tests (because it will break production!) It sounds like a really bad idea. It's like saying: When juggling, it's best to start with knifes, because every mistake will mean you're going to cut yourself, and you will quickly learn not to make mistakes.
  - virgilp 6y ago
    Except the "production" when you start is really a pre-production environment, meaning you won't _really_ cut yourself. To modify your analogy - which strategy is better: - start with blunt knives, realize the opportunities to hurt yourself, put safeguards in place (e.g. use protective gloves) OR - practice juggling balls until you get really good at it, then switch to knives for the real show.
borplk 6y ago
Each deploy containing fewer changes makes sense however it's not the only reason. Another significant reason is that more frequent deploys means more opportunities to identify and resolve any issues with the process. It becomes like a well-oiled machine. On the other hand when a process is run infrequently you tend to get surprised each time because something has changed or broken since the last time. Or you just simply haven't done the thing enough to observe the possible issues so every six months you discover and re-discover the problems.
Leherenn 6y ago
It only works if you control the deployment (like in a webapp). Try pushing firmware updates to non-internet connected devices every day or week, and you will see a lot of customer backlash.

jagged-chisel 6y ago

It’s true that deploying frequently reduce the surface of potential new problems per release. But to make it work flawlessly (not perfectly, but approaching) you do need some engineering rigor in the product.

lostcolony 6y ago

It's generally true, yes. While manual testing changes the equation some (the testing effort is the same for each release; the more you can automate that, the greater gains), how easy to -fix- the issue becomes more complicated the more changes there are. 1 PR, you know what introduced the breakage. 30 far reaching ones, with interesting overlap between them? Good luck.

seanwilson 6y ago
> 1 PR, you know what introduced the breakage. 30? Good luck. To be clearer, I mean there would be say 30 pull requests merged into "develop" and they would only be merged into "master" and pushed live after detailed manual testing. This is versus merging each pull request directly into "master" and going live immediately, where you wouldn't have time to do detailed manual testing every merge. You would still have the git history in both approaches to debug.
- lostcolony 6y ago
  Right. I'm saying there are two different things here. Regardless of your merge strategy. You either test after each PR merge, or you test only when you 'craft a release'. Doesn't matter whether this merge is into a develop branch or straight into master; it's solely a question of when you test it. If you manually test, testing only a batch of changes, together, is obviously easier, from a testing perspective. However, the effort to figure out and fix the issue can be very large. Plus, after fixing that one issue, you have to retest everything. So every bug requires retesting in any case (so while it's still a lower testing burden, unless you introduced 30+ unrelated issues, it's not as low as you think it might be on the surface). Compared with testing (and at that point you might as well release if your pipeline and org let you) with each PR - any manual testing effort is obviously higher due to testing so often (whereas automated is no extra effort), but, figuring out what caused the breakage, and addressing it, is much, much easier, order of magnitude easier, to figure out, and fixing it is much less likely to introduce new issues. Which is more to the original point; catching a bug earlier, allows the fix to be more targeted, which makes it more likely to be right, which reduces the likelihood of it making it to prod. There is also, to the original point, a statistical thing to consider. Let's say in 30 PRs, there is one that introduces a bug. Well, if QA is testing the one PR that broke something, they're more likely to spot it, since they're focusing on that PR. They're less likely if it's one of thirty sets of changes. But beyond that, let's say, in both situations, QA misses it. If you have one deploy, you've broken prod 100% of your releases (1 of 1 release). If you have 30 deploys, you've broken prod only ~3% of your releases. So immediately the original statement is validated; if your testing effort is the same (big if!), more frequent releases, with smaller changes, means the same or better percentage of deploying working code. Also, if you have to rollback with one mega release, you lose all 29 other changes; if you have to rollback with separate changes, you lose no other changes.

JamesBarney 6y ago

It's true in the sense that # of bugs or downtime per release is smaller. But that's not what clients care about at all. They care about total number of bugs and downtime. In addition this completely ignores that if you spend the same amount of time testing your testing per release drops when you increase the number of deployments.

johnmaguire2013 6y ago
> In addition this completely ignores that if you spend the same amount of time testing your testing per release drops when you increase the number of deployments. No, you just automate your testing.
- JamesBarney 6y ago
  I've never worked anywhere where 100% of testing was automated.
  - johnmaguire2013 6y ago
    In practice, automating around 90% of it is enough. This does explain why it didn't work for you though.

cortesoft 6y ago

If your code is that brittle, than sure.

shepherdjerred 6y ago

I would hate to work somewhere that my changes are deployed within days of writing my code. CI/CD lets you iterate faster and it encourages best practices like proper testing. That said, if you do CI/CD without proper testing then you're going to break production.

seanwilson 6y ago
Wait until you have to work with clients that don't even have staging environments let alone automated testing. Source control, CI/CD, automated testing, pull requests, code reviews, feature flags, linters, good logging etc. might be the norm on HN but not everywhere, especially companies where tech isn't the focus. It can also be a tough sell to make process improvements like this when the old way got them this far and they want new features now. I've worked with clients that don't even use source control, do all edits on production and there's no way setup to develop locally.
- pletnes 6y ago
  Although my current day job is very professional about this, I’ve seen the other side, too. «We have some code from 5-10 y back, can you compile it?» and only half the files are there. It would be great if HN had some content on «moving from random files on a network drive to git and basic automated testing» - it’s not an easy job and others would benefit from such writeups.
  - oblio 6y ago
    Joel on Software: go through the "Rock Star Developer" section of his archive.
lmm 6y ago
> I would hate to work somewhere that my changes are deployed within days of writing my code. Did you miss a "not" here? For me an environment where my changes are deployed and used quickly is a lot more fulfilling than one where I work on something that may only see the light of day weeks or months down the line.

lalaithion 6y ago

If you have decent automated testing, canarying, and monitoring, then yeah, releasing as frequently as possible means that usually a breakage is immediately narrowed down to 2-3 CLs. We release once every 2 hours, because it takes about 2 hours to cut, test, and slowly deploy one release. When we have a bug, we usually notice during the canarying process, and then roll back and fix it, and figure out how to test for that case.

virgilp 6y ago

That's the whole idea of CD: if you set up your project from the start so that deploys go to production, it changes the team culture and the way of working. People don't commit code without any tests (because it will break production!) and conversely don't stress over manually testing dozens of scenarios, some of which were "one-offs" that may become forgotten - it's "normal" that your code gets released, so the release is no longer a large, time-consuming and stressful event. If you start with a project where code is brittle and there are no tests, it is indeed extremely hard and time consuming to move to CI/CD. Doesn't mean it's not a good idea, just that it won't be easy and it'll take time (you have to build the whole infrastructure of automated testing, canary deployments, etc. - but, it's a good idea to do that anyway! It'll gradually improve the velocity. The alternative is that you end up in a world where everybody is afraid of making changes and the simplest requirement ends up getting estimates like "3 months of work".

js8 6y ago
> People don't commit code without any tests (because it will break production!) It sounds like a really bad idea. It's like saying: When juggling, it's best to start with knifes, because every mistake will mean you're going to cut yourself, and you will quickly learn not to make mistakes.
- virgilp 6y ago
  Except the "production" when you start is really a pre-production environment, meaning you won't _really_ cut yourself. To modify your analogy - which strategy is better: - start with blunt knives, realize the opportunities to hurt yourself, put safeguards in place (e.g. use protective gloves) OR - practice juggling balls until you get really good at it, then switch to knives for the real show.

borplk 6y ago

Each deploy containing fewer changes makes sense however it's not the only reason. Another significant reason is that more frequent deploys means more opportunities to identify and resolve any issues with the process. It becomes like a well-oiled machine. On the other hand when a process is run infrequently you tend to get surprised each time because something has changed or broken since the last time. Or you just simply haven't done the thing enough to observe the possible issues so every six months you discover and re-discover the problems.

Leherenn 6y ago

It only works if you control the deployment (like in a webapp). Try pushing firmware updates to non-internet connected devices every day or week, and you will see a lot of customer backlash.