4 ms·

One deploy per commit for a product with large team sounds insane - we'd easily have 50 commits a day for some hosted components. Even deploying this often to t

One deploy per commit for a product with large team sounds insane - we'd easily have 50 commits a day for some hosted components. Even deploying this often to the non-prod environments is problematic (it takes a fair bit longer than 1/50th a day for a start!), but doing that to your customer-facing production sites sounds bonkers - what class of problems is there that that's the best way to prevent them?

theptip 4y ago
Yes it's aspirational, and of course you have to do batching when the rate of commits gets too high. Many companies don't actually end up at the end of the spectrum with full continuous delivery; most of the value is captured by building what's needed _to be able to_ if you chose to, and doing say daily or hourly automated deploys. If you're not familiar with the general arguments for why CD might be desirable, a good place to start would be the books Continuous Delivery [https://smile.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley-ebook/dp/B003YMNVC0/ https://smile.amazon.com/Continuous-Delivery-Deployment-Auto...] or Accelerate [https://smile.amazon.com/Accelerate-Software-Performing-Technology-Organizations-ebook/dp/B07B9F83WM/ https://smile.amazon.com/Accelerate-Software-Performing-Tech...]. The TLDR would be that most deploy problems in practice turn out to be because the process is manual, infrequently-used, and/or unreliable, and by practicing "if it hurts, do it more often" you are forced to iron out those kinks. For example you can't deploy automatically without a good acceptance/smoke test suite, and without solid DB migration testing/linting/practices, to name just two of many. You could just build all these things without trying to do CD, but in practice it seems that it's quite hard for organizations to prioritize this work in isolation, and the goal of CD seems to provide a fairly clear map to a much higher-quality deploy process. The practice of deploying more frequently to flush out issues, is also quite unintuitive as you note. Here's a (probably out-of-date) article on how FB did deploys in 2017: https://www.infoq.com/news/2017/09/facebook-release-scale/ https://www.infoq.com/news/2017/09/facebook-release-scale/ -- batched commits and staged canary rollouts automatically cut and released every few hours.
- wizofaus 4y ago
  Yes I do get the general argument for CD (though I have issues with how universally it can be applied based on the sort of BVT I've had to do for SaaS platforms I worked on in the past, where it just wasn't possible to safely release new versions of key components without needing several days' worth of testing, most of which was waiting for verification that interactions with banks etc. had worked as expected).
  - theptip 4y ago
    Agreed, like almost everything it’s not universally applicable. If you have external contracts that are onerous to validate then it can be quite expensive to build high-fidelity internal representations of those contracts. Could still be ROI positive to do so! In most cases the contracts are either internal or simple (non-temporal) external contracts like a REST API, so most orgs benefit from moving in this direction.
YetAnotherNick 4y ago
> customer-facing production sites sounds bonkers We came from very different lands I guess. Batching 50 commits with no easy way of knowing which commit caused the issue seems scarier to me, in case of an issue. And if you code correctly and have warmup requests, your deployment could be without any downtime. Obviously there will be some extra resource usage because of unready resources.
- wizofaus 4y ago
  But if you're doing 50+ deploys a day, how can there possibly enough time to know which commit caused whatever issue that gets noticed anyway? If those 50 deploys are done during regular business hours that's 9 minutes per deploy on average!
- YetAnotherNick 4y ago
  Reply to wizofaus: 9 minutes is more than much more than enough to revert, at least in our setup. Specially if some automated alert triggers, then we could revert well within a minute.
  - wizofaus 4y ago
    I can't even begin to think how to get there, at least using CloudFormation.
    - theptip 4y ago
      More advanced/modern tool chains like k8s can help; rollout can be done in minutes. Also makes it easier to do blue/green deploys if you’re serving high throughput services where you can get an error signal from upgrading a small fraction of your service workloads.
  - pjerem 4y ago
    That’s what staging environments are for : deploying every commit to check everything is fine, doing the QA and get the alerts. Doing this in production is being disrespectful of your users. Users are not waiting for your PRs to be approved and merged, they can be as delighted by your new button’s shade of lightgrey if it is released some days (even weeks !) after.

theptip 4y ago

Yes it's aspirational, and of course you have to do batching when the rate of commits gets too high. Many companies don't actually end up at the end of the spectrum with full continuous delivery; most of the value is captured by building what's needed _to be able to_ if you chose to, and doing say daily or hourly automated deploys. If you're not familiar with the general arguments for why CD might be desirable, a good place to start would be the books Continuous Delivery [https://smile.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley-ebook/dp/B003YMNVC0/ https://smile.amazon.com/Continuous-Delivery-Deployment-Auto...] or Accelerate [https://smile.amazon.com/Accelerate-Software-Performing-Technology-Organizations-ebook/dp/B07B9F83WM/ https://smile.amazon.com/Accelerate-Software-Performing-Tech...]. The TLDR would be that most deploy problems in practice turn out to be because the process is manual, infrequently-used, and/or unreliable, and by practicing "if it hurts, do it more often" you are forced to iron out those kinks. For example you can't deploy automatically without a good acceptance/smoke test suite, and without solid DB migration testing/linting/practices, to name just two of many. You could just build all these things without trying to do CD, but in practice it seems that it's quite hard for organizations to prioritize this work in isolation, and the goal of CD seems to provide a fairly clear map to a much higher-quality deploy process. The practice of deploying more frequently to flush out issues, is also quite unintuitive as you note. Here's a (probably out-of-date) article on how FB did deploys in 2017: https://www.infoq.com/news/2017/09/facebook-release-scale/ https://www.infoq.com/news/2017/09/facebook-release-scale/ -- batched commits and staged canary rollouts automatically cut and released every few hours.

wizofaus 4y ago
Yes I do get the general argument for CD (though I have issues with how universally it can be applied based on the sort of BVT I've had to do for SaaS platforms I worked on in the past, where it just wasn't possible to safely release new versions of key components without needing several days' worth of testing, most of which was waiting for verification that interactions with banks etc. had worked as expected).
- theptip 4y ago
  Agreed, like almost everything it’s not universally applicable. If you have external contracts that are onerous to validate then it can be quite expensive to build high-fidelity internal representations of those contracts. Could still be ROI positive to do so! In most cases the contracts are either internal or simple (non-temporal) external contracts like a REST API, so most orgs benefit from moving in this direction.

YetAnotherNick 4y ago

> customer-facing production sites sounds bonkers We came from very different lands I guess. Batching 50 commits with no easy way of knowing which commit caused the issue seems scarier to me, in case of an issue. And if you code correctly and have warmup requests, your deployment could be without any downtime. Obviously there will be some extra resource usage because of unready resources.

wizofaus 4y ago
But if you're doing 50+ deploys a day, how can there possibly enough time to know which commit caused whatever issue that gets noticed anyway? If those 50 deploys are done during regular business hours that's 9 minutes per deploy on average!
YetAnotherNick 4y ago
Reply to wizofaus: 9 minutes is more than much more than enough to revert, at least in our setup. Specially if some automated alert triggers, then we could revert well within a minute.
- wizofaus 4y ago
  I can't even begin to think how to get there, at least using CloudFormation.
  - theptip 4y ago
    More advanced/modern tool chains like k8s can help; rollout can be done in minutes. Also makes it easier to do blue/green deploys if you’re serving high throughput services where you can get an error signal from upgrading a small fraction of your service workloads.
- pjerem 4y ago
  That’s what staging environments are for : deploying every commit to check everything is fine, doing the QA and get the alerts. Doing this in production is being disrespectful of your users. Users are not waiting for your PRs to be approved and merged, they can be as delighted by your new button’s shade of lightgrey if it is released some days (even weeks !) after.