5 ms·

I've come to question releasing often as a result.

jon-wood 10y ago
I can see the argument that if releasing causes things to break then don't release so frequently, but in practice the end result of that is lots of things breaking at once and having to unpick everything. Debugging is much easier if you're debugging a single change fresh in your mind.
- bryanlarsen 10y ago
  Yeah, debugging 2 bugs that are shadowing each other often takes several orders of magnitude longer to debug than just a single issue.
- Thrillington 10y ago
  While having tooling for tight debugging loops allows short release cycles, it leaves that decision up to the business
- asimuvPR 10y ago
  You are right. Making small changes that can be isolated for debugging purposes is a good approach. What I mean is that we should always question "best" practices and how we apply them to our development process. These days there is a tendency to drink the kool-aid (guilty of this as well). DevOps is something that we are still learning and developing as a profession.
  - dozzie 10y ago
    > DevOps is something that we are still learning and developing as a profession. Not really. DevOps is simply a modernish label for system administration that we have for dozens years already.
    - the_other 10y ago
      Isn't that just "Ops"?
      - dozzie 10y ago
        My point exactly.
        aurelianito 10y ago
        Yes, but now we actually try to automate in a systematic way; as opposed to a single admin hacking some custom shell scripts. That's what the dev in devop mean. And that's new, at least it was not a common activity in the XX century.
        dozzie 10y ago
        If by "systematic way" you mean a group of programmers clueless about system administration hacking together some custom Ansible scripts, then I wouldn't call it progress. Sysadmins have tools to automate their work for a long, long time (cfengine, bcfg2, even Puppet and Chef predates DevOps hype). DevOps didn't bring anything new to the table.
        aurelianito 10y ago
        Is not about doing it right or wrong. Is about the automation that is reused cross-project and the fact that a lot of things that used to require a sysadmin now we automated its job away. If developers implement it correctly or poorly is a different issue. It is also not about hype (or not). I do agree that the name is posterior to the beginning of the practice, but it is the name that we have.
        dozzie 10y ago
        Being a sysadmin was always about automation. DevOps brought nothing to the table about that. Neither tools nor paradigm. And the name is just another one for "administering systems", if we keep what you seem to mean by DevOps. It's still no progress, hacky scripts written by sysadmins or similarly hacky scripts written by programmers, and systematic way of automating tasks was available to sysadmins and was used by them for a long time. DevOps brought nothing new to the table.
zaroth 10y ago
It seems like so many "best practices" are really thinly veiled attempts at exploding complexity with only tenuous potential business advantages. We create ourselves so many of the problems we are paid to solve.
- mLuby 10y ago
  "We create ourselves so many of the problems we are paid to solve" Sounds lucrative ;)
  - jessaustin 10y ago
    It works for doctors and lawyers; why shouldn't software developers try it?
- thomaslee 10y ago
  > It seems like so many "best practices" are really thinly veiled attempts at exploding complexity I'll bite. :D It's a bit more nuanced than that IMO, the "deploy often" mantra is only as good as the process around release + deploy. If you half-ass testing and push to production without without a process for verification -- or if, say, your deploy process is half-baked, or your staging environment is worthless -- you can probably expect "interesting" production deploys on a pretty regular basis. As much as we'd like to pretend we're all good engineers, this happens more often than you'd expect -- even with good engineers: at some point a company transitions from scrappy startup to a shambling beast, and the things that used to work for a scrappy startup (like skimping on testing and dealing with failures in production) are insufficient when you've got more eyes on the product. Further, the engineering culture remains stuck in "scrappy startup" mode long after the shambling has begun. And all that's ignoring the fact that less frequent deploys with more changes have their own set of problems. We actually got to a point with deploys of a certain distributed system such that we were terrified if we had more than a few days worth of changes queued up. So many things that could go wrong! :) > We create ourselves so many of the problems we are paid to solve. This, on the other hand, I completely 100% agree with: if not us, then who? :) EDIT: minor formatting change
cortesoft 10y ago
The breakage rate per new feature is fairly constant; if you release 7 new features once a week or 1 new feature once a day, you will have the same number of issues. The question then is; is it easier to deal with all the issues at once, or a smaller number of issues every day?
- danek 10y ago
  In addition to that, I would also worry about interactions between issues. I tend to lean towards spreading out the issues over time to make the eventual diagnosis easier. Occasionally we'll have a problem where we cannot deploy to production for several days (normally it's once a day). A massive inventory of ready-to-deploy features builds up. When we do finally deploy, this deluge of features and fixes creates new issues that force a rollback, delaying the deploy even longer...
  - pc86 10y ago
    Sounds like a testing/UAT issue.
  - cortesoft 10y ago
    Absolutely. Confounding issues are the biggest problem, even more so than sheer volume in a small time frame.
msoad 10y ago
You can roll back a small release that broke the world. A
AYBABTME 10y ago
Or the opposite: embrace failure, practice resiliency, work with it instead of against it!
awj 10y ago
Funny, I've taken it as justification for releasing often. If I have a hard time changing one thing without breaking the service, it's nearly impossible to change a hundred things without breaking something. Since it's a given that I will have to change things, I'll try to stick to a scope where I stand a chance of doing so successfully.

I can see the argument that if releasing causes things to break then don't release so frequently, but in practice the end result of that is lots of things breaking at once and having to unpick everything. Debugging is much easier if you're debugging a single change fresh in your mind.

bryanlarsen 10y ago
Yeah, debugging 2 bugs that are shadowing each other often takes several orders of magnitude longer to debug than just a single issue.
Thrillington 10y ago
While having tooling for tight debugging loops allows short release cycles, it leaves that decision up to the business
asimuvPR 10y ago
You are right. Making small changes that can be isolated for debugging purposes is a good approach. What I mean is that we should always question "best" practices and how we apply them to our development process. These days there is a tendency to drink the kool-aid (guilty of this as well). DevOps is something that we are still learning and developing as a profession.
- dozzie 10y ago
  > DevOps is something that we are still learning and developing as a profession. Not really. DevOps is simply a modernish label for system administration that we have for dozens years already.
  - the_other 10y ago
    Isn't that just "Ops"?
    - dozzie 10y ago
      My point exactly.
      - aurelianito 10y ago
        Yes, but now we actually try to automate in a systematic way; as opposed to a single admin hacking some custom shell scripts. That's what the dev in devop mean. And that's new, at least it was not a common activity in the XX century.
        dozzie 10y ago
        If by "systematic way" you mean a group of programmers clueless about system administration hacking together some custom Ansible scripts, then I wouldn't call it progress. Sysadmins have tools to automate their work for a long, long time (cfengine, bcfg2, even Puppet and Chef predates DevOps hype). DevOps didn't bring anything new to the table.
        aurelianito 10y ago
        Is not about doing it right or wrong. Is about the automation that is reused cross-project and the fact that a lot of things that used to require a sysadmin now we automated its job away. If developers implement it correctly or poorly is a different issue. It is also not about hype (or not). I do agree that the name is posterior to the beginning of the practice, but it is the name that we have.
        dozzie 10y ago
        Being a sysadmin was always about automation. DevOps brought nothing to the table about that. Neither tools nor paradigm. And the name is just another one for "administering systems", if we keep what you seem to mean by DevOps. It's still no progress, hacky scripts written by sysadmins or similarly hacky scripts written by programmers, and systematic way of automating tasks was available to sysadmins and was used by them for a long time. DevOps brought nothing new to the table.

zaroth 10y ago

It seems like so many "best practices" are really thinly veiled attempts at exploding complexity with only tenuous potential business advantages. We create ourselves so many of the problems we are paid to solve.

mLuby 10y ago
"We create ourselves so many of the problems we are paid to solve" Sounds lucrative ;)
- jessaustin 10y ago
  It works for doctors and lawyers; why shouldn't software developers try it?
thomaslee 10y ago
> It seems like so many "best practices" are really thinly veiled attempts at exploding complexity I'll bite. :D It's a bit more nuanced than that IMO, the "deploy often" mantra is only as good as the process around release + deploy. If you half-ass testing and push to production without without a process for verification -- or if, say, your deploy process is half-baked, or your staging environment is worthless -- you can probably expect "interesting" production deploys on a pretty regular basis. As much as we'd like to pretend we're all good engineers, this happens more often than you'd expect -- even with good engineers: at some point a company transitions from scrappy startup to a shambling beast, and the things that used to work for a scrappy startup (like skimping on testing and dealing with failures in production) are insufficient when you've got more eyes on the product. Further, the engineering culture remains stuck in "scrappy startup" mode long after the shambling has begun. And all that's ignoring the fact that less frequent deploys with more changes have their own set of problems. We actually got to a point with deploys of a certain distributed system such that we were terrified if we had more than a few days worth of changes queued up. So many things that could go wrong! :) > We create ourselves so many of the problems we are paid to solve. This, on the other hand, I completely 100% agree with: if not us, then who? :) EDIT: minor formatting change

cortesoft 10y ago

The breakage rate per new feature is fairly constant; if you release 7 new features once a week or 1 new feature once a day, you will have the same number of issues. The question then is; is it easier to deal with all the issues at once, or a smaller number of issues every day?

danek 10y ago
In addition to that, I would also worry about interactions between issues. I tend to lean towards spreading out the issues over time to make the eventual diagnosis easier. Occasionally we'll have a problem where we cannot deploy to production for several days (normally it's once a day). A massive inventory of ready-to-deploy features builds up. When we do finally deploy, this deluge of features and fixes creates new issues that force a rollback, delaying the deploy even longer...
- pc86 10y ago
  Sounds like a testing/UAT issue.
- cortesoft 10y ago
  Absolutely. Confounding issues are the biggest problem, even more so than sheer volume in a small time frame.

msoad 10y ago

You can roll back a small release that broke the world. A

AYBABTME 10y ago

Or the opposite: embrace failure, practice resiliency, work with it instead of against it!

awj 10y ago

Funny, I've taken it as justification for releasing often. If I have a hard time changing one thing without breaking the service, it's nearly impossible to change a hundred things without breaking something. Since it's a given that I will have to change things, I'll try to stick to a scope where I stand a chance of doing so successfully.