3 ms·

I really enjoyed this article. As an industry, when it comes to something essential like source control, we seem to have converged to a common set of practices

I really enjoyed this article. As an industry, when it comes to something essential like source control, we seem to have converged to a common set of practices and workflows. Deployment is arguably just as important, but I think the practices are very different on different teams. This article is like a more practical version of Continuous Delivery. Three areas that I think would have been worth including: 1. Pre-production. You deploy to test & other pre-prod environments more often than prod. They should use the same scripts/tools/processes as production deployments, only with different permissions. 2. Configuration. Test and production environments will always have different config settings, so no team will ever be able to deploy to more than one environment without encountering this problem. I think there's still an open question around whether those configuration settings should live in the same source control as the code, in a different source control repository, or a dedicated system. Source control systems and sensitive values (passwords, API keys, etc.) don't always mix. 3. Build your binaries once. The article is more focussed on dynamic languages, but for compiled languages, I think this is important. If you branch, compile, deploy to test, test it, get the all clear, then compile again and deploy to production, there's a lot of opportunity for differences between what you tested and what goes to production to sneak in. In fact even for dynamic languages, this might be a valuable practice. What if the JS minifier on one build server is different to another, and the deployed script ends up being different in production to what was tested. Disclaimer: I'm the founder of Octopus Deploy, and these practices might be biased towards enterprisey .NET/on-premises deployments rather than cloud hosted, dynamic language projects.

Cheezmeister 11y ago
> 3. Build your binaries once. Assuming your builds are solidly reproducible like they should be, how do such differences "sneak in"? Granted, truly reproducible builds in the first place are Really Hard(tm).
- paulstovell 11y ago
  We put an explanation here (hope it's OK to post the link): https://octopus.com/blog/build-your-binaries-once https://octopus.com/blog/build-your-binaries-once A real world example of this is that when .NET 4.5 shipped, the compilers in .NET 4.5 for 4.0 code produced different output than the .NET 4.0 compilers would have. So installing a system-level update on Windows on a build server would mean Test and Production got different results: http://blog.marcgravell.com/2012/09/iterator-blocks-missing-methods-and-net.html http://blog.marcgravell.com/2012/09/iterator-blocks-missing-... Also, releases move through environments at different times: Monday: 1.0 goes to Test Tuesday: 1.1 goes to Test Wednesday: 1.0 goes to Prod Between 1.0 and 1.1, perhaps you updated Node, or went to Python3 - so your build server had to be updated. Reproducing Monday's build is going to be more difficult the more time that goes by. As you said, truly reproducible builds are hard. Why not just zip the artifacts and use the same files, instead of rebuilding?
  - manojlds 11y ago
    Docker for CI will help. Update your base image when you upgrade Node/Python.
Spearchucker 11y ago
1. Pre-production Pre-production should test production in a non-production environment. Specifically, if a system connects to, say, SalesForce, and you've been using a local sandboxed instance of SalesForce in dev and UAT, the pre-prod should connect to the real, live SalesForce. I have a pre-prod environment for my current project, and it's useless because org policy states that only prod environments can connect to live instances of anything. Completely invalidates the reason for having pre-prod. 2. Configuration ...will always have different config settings... Yes, aside from pre-prod. Pre-prod and prod MUST be mirror images, INCLUDING config settings. Identical. The same in every way except for host names.
- UK-AL 11y ago
  We use octopus deploy(Great software btw), most people use configuration stuff mostly for connection strings. Everything else is the same.
jmsb 11y ago
> 1. Pre-production It interests me in how many combinations of pre-prod environments exist, as I (naively) expected this to be standardized, but it's not (e.g. https://en.wikipedia.org/wiki/Deployment_environment#Environments https://en.wikipedia.org/wiki/Deployment_environment#Environ... ). I can see the need for two vs four environments for handling different stages of the life-cycle depending on business needs (and budget). Addressing how to choose the right set of environments for a given company/product would be useful. I also want to say that your product is phenomenal and it has improved the quality of our shop's build and deploy pipeline by an order of magnitude. Thank you.

Cheezmeister 11y ago

> 3. Build your binaries once. Assuming your builds are solidly reproducible like they should be, how do such differences "sneak in"? Granted, truly reproducible builds in the first place are Really Hard(tm).

paulstovell 11y ago
We put an explanation here (hope it's OK to post the link): https://octopus.com/blog/build-your-binaries-once https://octopus.com/blog/build-your-binaries-once A real world example of this is that when .NET 4.5 shipped, the compilers in .NET 4.5 for 4.0 code produced different output than the .NET 4.0 compilers would have. So installing a system-level update on Windows on a build server would mean Test and Production got different results: http://blog.marcgravell.com/2012/09/iterator-blocks-missing-methods-and-net.html http://blog.marcgravell.com/2012/09/iterator-blocks-missing-... Also, releases move through environments at different times: Monday: 1.0 goes to Test Tuesday: 1.1 goes to Test Wednesday: 1.0 goes to Prod Between 1.0 and 1.1, perhaps you updated Node, or went to Python3 - so your build server had to be updated. Reproducing Monday's build is going to be more difficult the more time that goes by. As you said, truly reproducible builds are hard. Why not just zip the artifacts and use the same files, instead of rebuilding?
- manojlds 11y ago
  Docker for CI will help. Update your base image when you upgrade Node/Python.

Spearchucker 11y ago

1. Pre-production Pre-production should test production in a non-production environment. Specifically, if a system connects to, say, SalesForce, and you've been using a local sandboxed instance of SalesForce in dev and UAT, the pre-prod should connect to the real, live SalesForce. I have a pre-prod environment for my current project, and it's useless because org policy states that only prod environments can connect to live instances of anything. Completely invalidates the reason for having pre-prod. 2. Configuration ...will always have different config settings... Yes, aside from pre-prod. Pre-prod and prod MUST be mirror images, INCLUDING config settings. Identical. The same in every way except for host names.

UK-AL 11y ago
We use octopus deploy(Great software btw), most people use configuration stuff mostly for connection strings. Everything else is the same.

jmsb 11y ago

> 1. Pre-production It interests me in how many combinations of pre-prod environments exist, as I (naively) expected this to be standardized, but it's not (e.g. https://en.wikipedia.org/wiki/Deployment_environment#Environments https://en.wikipedia.org/wiki/Deployment_environment#Environ... ). I can see the need for two vs four environments for handling different stages of the life-cycle depending on business needs (and budget). Addressing how to choose the right set of environments for a given company/product would be useful. I also want to say that your product is phenomenal and it has improved the quality of our shop's build and deploy pipeline by an order of magnitude. Thank you.