5 ms·

I asked how to roll back the changes. He said "we don't roll back, we fix the code." Not the best idea. I don't debug as well under the extreme pressure of my

I asked how to roll back the changes. He said "we don't roll back, we fix the code." Not the best idea. I don't debug as well under the extreme pressure of my site being broken with the clock ticking.

rue 16y ago
Not the best idea, if you don't debug as well under such pressure. Some people do.
- Alex3917 16y ago
  Or, more commonly, people think they don't perform well under this kind of pressure, but they actually do. Usual scenario: Employee calls the CEO of the company in the middle of the night panicked, saying the site is down and there is no way they can fix the problem, they don't even have the right skill set. CEO tells the employee to calm down and do their best. Site is back online twelve minutes later.
  - ghurlman 16y ago
    More likely, the developer can handle the tech pressure, but having his/her manager FREAKING OUT and over his shoulder asking questions the whole time is what really causes the slowdown.
    - kellanem 16y ago
      Step 0. in getting started with continuous deployment is having an organization that doesn't lose it's mind every time there's a blip. That often means you need managers who are especially good at (or at least dedicated to) deflecting the freak out/crazy. CD won't increase your changed related incidents, but to paraphrase the IBM parable, "no one ever got fired, for doing quarterly releases, and heavy QA."
- ZoFreX 16y ago
  I used to code for a website handling about 25,000 logged in users per day, and there was a post-commit hook on SVN which pushed changes live, immediately. Working on that taught me a lot (mostly how not* to do things), one thing I learnt is that I code much, much faster under that kind of pressure. * Note: I am not endorsing this as a setup, it's crazy. I don't think I've ever seen such high staff turnover.
bdotdub 16y ago
When you know you can't really rollback, I'm sure you're helluva lot more sure that your code won't take down the site
mrkurt 16y ago
You'd probably find it easier when the deployed changes were extremely small. Our last automated deploy was a single line change. Most are bigger, but not huge. Also, while they don't do full rollbacks, I suspect more than one fix has been "remove the offending code until we can figure out what's wrong".
- Bolyuba 16y ago
  Agree, it seems to me that an emphasis is on time production deployment takes rather than rollback vs fix
- rm-rf 16y ago
  I've seen single line changes cause data loss, corruption, system outages, remote root exploits... I'm not sure that the number of lines makes the change more or less risky.
  - ichverstehe 16y ago
    And how exactly does this relate to the way they deploy their code? As far as I can tell, they actually review code before marking it ready for deploy. That kind of changes would be an issue with "usual" large batch deploys as well.
  - chaz 16y ago
    I've also seen many devs look at a broken release and instantly realize their mistake. A forgotten production config, a hard coded variable, or an empty cache. If code is in fact reviewed before going to production, the risk is significantly lower.
  - cool-RR 16y ago
    Nice choice of username, by the way.
  - MartinCron 16y ago
    The safest change to make to a stable system is the smallest change possible.
Groxx 16y ago
I'd bet it's semantics. Rolling back the code-base would mean everyone else would need to roll back theirs, and you'd lose history. It's easier across the board to simply re-write what it was - a "roll forward", if you will.
neilk 16y ago
It's not always like that. John Allspaw's answer is correct in the sense that actual rollbacks aren't done -- the fix will be a new push -- but sometimes, before you do that, you will just turn the new feature off in production. This is possible because many new behaviour is controlled by "feature flags" which are associated with the server configuration. So you have the benefit of everybody, developers and production, staying very close to trunk, but still being able to have radically different behavior on development machines versus production. That said, I have participated in debugging sessions on a product which used continuous deployment, and they can indeed be nerve-wracking. Personally I wouldn't want to use CD by itself, without a good culture of code review and a great test suite.
- InclinedPlane 16y ago
  Feature flags are a very powerful feature, especially for services. Many of the most successful web companies (amazon, google, facebook, twitter) have entire frameworks and infrastructure dedicated to supporting the design pattern.

rue 16y ago

Not the best idea, if you don't debug as well under such pressure. Some people do.

Alex3917 16y ago
Or, more commonly, people think they don't perform well under this kind of pressure, but they actually do. Usual scenario: Employee calls the CEO of the company in the middle of the night panicked, saying the site is down and there is no way they can fix the problem, they don't even have the right skill set. CEO tells the employee to calm down and do their best. Site is back online twelve minutes later.
- ghurlman 16y ago
  More likely, the developer can handle the tech pressure, but having his/her manager FREAKING OUT and over his shoulder asking questions the whole time is what really causes the slowdown.
  - kellanem 16y ago
    Step 0. in getting started with continuous deployment is having an organization that doesn't lose it's mind every time there's a blip. That often means you need managers who are especially good at (or at least dedicated to) deflecting the freak out/crazy. CD won't increase your changed related incidents, but to paraphrase the IBM parable, "no one ever got fired, for doing quarterly releases, and heavy QA."
ZoFreX 16y ago
I used to code for a website handling about 25,000 logged in users per day, and there was a post-commit hook on SVN which pushed changes live, immediately. Working on that taught me a lot (mostly how not* to do things), one thing I learnt is that I code much, much faster under that kind of pressure. * Note: I am not endorsing this as a setup, it's crazy. I don't think I've ever seen such high staff turnover.

bdotdub 16y ago

When you know you can't really rollback, I'm sure you're helluva lot more sure that your code won't take down the site

mrkurt 16y ago

You'd probably find it easier when the deployed changes were extremely small. Our last automated deploy was a single line change. Most are bigger, but not huge. Also, while they don't do full rollbacks, I suspect more than one fix has been "remove the offending code until we can figure out what's wrong".

Bolyuba 16y ago
Agree, it seems to me that an emphasis is on time production deployment takes rather than rollback vs fix
rm-rf 16y ago
I've seen single line changes cause data loss, corruption, system outages, remote root exploits... I'm not sure that the number of lines makes the change more or less risky.
- ichverstehe 16y ago
  And how exactly does this relate to the way they deploy their code? As far as I can tell, they actually review code before marking it ready for deploy. That kind of changes would be an issue with "usual" large batch deploys as well.
- chaz 16y ago
  I've also seen many devs look at a broken release and instantly realize their mistake. A forgotten production config, a hard coded variable, or an empty cache. If code is in fact reviewed before going to production, the risk is significantly lower.
- cool-RR 16y ago
  Nice choice of username, by the way.
- MartinCron 16y ago
  The safest change to make to a stable system is the smallest change possible.

Groxx 16y ago

I'd bet it's semantics. Rolling back the code-base would mean everyone else would need to roll back theirs, and you'd lose history. It's easier across the board to simply re-write what it was - a "roll forward", if you will.

neilk 16y ago

It's not always like that. John Allspaw's answer is correct in the sense that actual rollbacks aren't done -- the fix will be a new push -- but sometimes, before you do that, you will just turn the new feature off in production. This is possible because many new behaviour is controlled by "feature flags" which are associated with the server configuration. So you have the benefit of everybody, developers and production, staying very close to trunk, but still being able to have radically different behavior on development machines versus production. That said, I have participated in debugging sessions on a product which used continuous deployment, and they can indeed be nerve-wracking. Personally I wouldn't want to use CD by itself, without a good culture of code review and a great test suite.

InclinedPlane 16y ago
Feature flags are a very powerful feature, especially for services. Many of the most successful web companies (amazon, google, facebook, twitter) have entire frameworks and infrastructure dedicated to supporting the design pattern.