6 ms·

> In a company where you ship continuously, you see this and think “Simple – I’ll tweak a test and a regex pattern, get a quick code review from my buddy beside

by luckyt 9y ago

> In a company where you ship continuously, you see this and think “Simple – I’ll tweak a test and a regex pattern, get a quick code review from my buddy beside me, merge to mainline, and 1 minute later when it’s deployed to production, reply to the customer: ‘Sorry about this, it’s fixed now, thanks for letting us know.’” This sounds like a really bad idea. If it's not a critical bug impacting all your customers, do you have to fix it in 1 minute? Once your company gets bigger, you would want to do proper code review and QA, so your 'one-line fix' doesn't accidentally bring down the whole site.

zeckalpha 9y ago
Yes, because otherwise non-critical bugs pile up and hide critical bugs.
- pyrophane 9y ago
  Another solution to this is regular bug triage.
  - zeckalpha 9y ago
    Stale inventory is expensive.
blumomo 9y ago
In a perfect scenario you would have set up continuous integration that automatically runs all your unit, integration and end-to-end tests. Ideally your coverage with tests is so good that you can rely on a successful integration of the bug fix commit. This of course is a ideal scenario that you have to work on seriously to be very close to it.
toast0 9y ago
If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken. Then you can do the fix again, but with more care and testing. If all goes well, you only have to think about this issue for 10 minutes, and then get on with your life, you don't have to spend 10 minutes a day following it through the 2 week deployment process and only then notify the customer that they can finally use the product. Also, if you think a fix needs more care, the ability to ship quickly doesn't impede doing it slowly; but in many companies there is simply no ability to ship fast, regardless of the type of bug.
- emmab 9y ago
  > If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken. Unless it caused data corruption.
  - toast0 9y ago
    Yes, that's why I said usually. It's also why I said, you can take it slow if you're concerned. And something that would possibly cause data corruption is a good thing to take more time with.
BillinghamJ 9y ago
You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using that code-path - within a few hours or so. Takes a bit of infra effort, but for a company like Intercom, that’s very easily achievable.
- ben_jones 9y ago
  what would be some good resources to learn how to build such infrastructure? I've seen it as a feature of various platforms but would imagine it could become complex otherwise.
  - BillinghamJ 9y ago
    It wouldn’t be practical without some kind of underlying infrastructure orchestration system. Obviously some companies have gone the extra step to actually build these systems - e.g. Google made Kubernetes. For any not-huge companies, I think it pretty much just falls to choosing one of the many existing tools. Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.
  - toast0 9y ago
    The simple way to do this can apply if you're mostly doing things in the context of a user. Write the change to check if a hash of the user identifier is less than a target; increase the target over time.
- AznHisoka 9y ago
  this is easy and doable if your change is a front end change. good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.
  - greenleafjacob 9y ago
    If the cost of downtime is high enough you can mirror some or all traffic onto a shadow copy of the infrastructure.
wpietri 9y ago
Continuous deployment works differently. Your approach focuses on reducing MTBF; continuous deployment focused on reducing MTTR. For CD to work, you can't really depend on QA for regression testing. You need to have really good automated test suites. You still do want exploratory QA, but that doesn't have to happen before release. Code review works similarly. As your quote says, he did proper code review. It was a very small change, so it doesn't take long. Strong test coverage means code review is less about finding possible bugs and more about design issues. There's no urgency to getting those before release; people can watch changelogs just as well as they can service a review queue. And with frequent releases, developers return to important code often, meaning that design improvements happen over time.

by luckyt 9y ago

zeckalpha 9y ago

Yes, because otherwise non-critical bugs pile up and hide critical bugs.

pyrophane 9y ago
Another solution to this is regular bug triage.
- zeckalpha 9y ago
  Stale inventory is expensive.

blumomo 9y ago

In a perfect scenario you would have set up continuous integration that automatically runs all your unit, integration and end-to-end tests. Ideally your coverage with tests is so good that you can rely on a successful integration of the bug fix commit. This of course is a ideal scenario that you have to work on seriously to be very close to it.

toast0 9y ago

If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken. Then you can do the fix again, but with more care and testing. If all goes well, you only have to think about this issue for 10 minutes, and then get on with your life, you don't have to spend 10 minutes a day following it through the 2 week deployment process and only then notify the customer that they can finally use the product. Also, if you think a fix needs more care, the ability to ship quickly doesn't impede doing it slowly; but in many companies there is simply no ability to ship fast, regardless of the type of bug.

emmab 9y ago
> If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken. Unless it caused data corruption.
- toast0 9y ago
  Yes, that's why I said usually. It's also why I said, you can take it slow if you're concerned. And something that would possibly cause data corruption is a good thing to take more time with.

BillinghamJ 9y ago

You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using that code-path - within a few hours or so. Takes a bit of infra effort, but for a company like Intercom, that’s very easily achievable.

ben_jones 9y ago
what would be some good resources to learn how to build such infrastructure? I've seen it as a feature of various platforms but would imagine it could become complex otherwise.
- BillinghamJ 9y ago
  It wouldn’t be practical without some kind of underlying infrastructure orchestration system. Obviously some companies have gone the extra step to actually build these systems - e.g. Google made Kubernetes. For any not-huge companies, I think it pretty much just falls to choosing one of the many existing tools. Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.
- toast0 9y ago
  The simple way to do this can apply if you're mostly doing things in the context of a user. Write the change to check if a hash of the user identifier is less than a target; increase the target over time.
AznHisoka 9y ago
this is easy and doable if your change is a front end change. good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.
- greenleafjacob 9y ago
  If the cost of downtime is high enough you can mirror some or all traffic onto a shadow copy of the infrastructure.

wpietri 9y ago

Continuous deployment works differently. Your approach focuses on reducing MTBF; continuous deployment focused on reducing MTTR. For CD to work, you can't really depend on QA for regression testing. You need to have really good automated test suites. You still do want exploratory QA, but that doesn't have to happen before release. Code review works similarly. As your quote says, he did proper code review. It was a very small change, so it doesn't take long. Strong test coverage means code review is less about finding possible bugs and more about design issues. There's no urgency to getting those before release; people can watch changelogs just as well as they can service a review queue. And with frequent releases, developers return to important code often, meaning that design improvements happen over time.