5 ms·
If you are doing devops, wouldn't deploying to 1 data center vs 20 be no more work? It's just a loop around a script.
by brianwawok 10y ago
If you are doing devops, wouldn't deploying to 1 data center vs 20 be no more work? It's just a loop around a script.
- danpalmer 10y agoAll it takes is for one server in one datacenter to be slightly different, or perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it, now you've got a deploy that will be a lot more complicated or error-prone than a loop around deployment to one location.
- icebraining 10y agoAll it takes is for one server in one datacenter to be slightly different But doesn't a global system still run on multiple DCs, at least for redundancy? perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it But if you have a single global system, you can't even make that decision. To be clear, I'm not arguing against it, I'm just trying to understand why, since my first instinct would be to divide geographically as well.
- danpalmer 10y ago> But doesn't a global system still run on multiple DCs, at least for redundancy? If you've got 2 levels of separation - servers and 'groups' (whether they are datacenters, or whatever) - you've got 2 levels at which that special casing needs to happen. If you only have 1 level - servers - i.e. one deployment, even if that's across multiple datacenters, you only have 1 place to special case. I'd say that's easier. > But if you have a single global system, you can't even make that decision. Good point, but my point was that it will be simpler and less error prone in general. You might not be able to push the bugfix, or you might have to risk the deploy globally, but I think either would be better in the long run for a simpler deployment. It is a trade-off though.
- brianwawok 10y ago> All it takes is for one server in one datacenter to be slightly different Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. > perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it Feature flags. Default off, but flip on a new path of code for a set of users. You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit). Not saying these are simple things to do, but if you are approaching servers with a devops mindsets, you literally should not care about number of servers or datacenters.
- matwood 10y ago> Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. Yep, this is change control 101. I used to have a boss who would go in and edit sprocs on the production server and never get them into source control. I finally just encrypted everything on the server so the only way to push out new code was to commit to source control. A sledge hammer yes, but sometimes it is required. Use Terraform or some other tool to codify your entire infrastructure. >You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit). This is a great point. At first it is terrifying, but then you realize that a deployment is so easy that bugs are generally not a big deal. Bugs happen, so the goal should be to shorten the time between bug found and fix deployed.
- krinchan 10y agoThis is so weird. I hear this all the time. At my place of employment, we all have the ssh keys into our EC2 instances, but no one configures them. Ever. Period. Those ssh keys are purely for either validating changes in a test environment (like to .ebextensions) or diagnosing production issues (why did Puma fall over this time? why isn't syslog output making it to loggly?). Of course, we lean heavily on Elastic Beanstalk; autoscaling regularly kills old instances daily since we scale from 2 to 18 and back to 2 instances in a 24 hour period across about 9 microservices. So, if this ssh into boxes and change things is common, it means people aren't doing auto-scaling? THAT is scary.
- matthewrudy 10y agoThere's the other aspect too, which is each country has their own cluster, so they can (and do) think of excuses to hold back updates just for them. So software gets out of sync, the mobile app releases need to take that into account too.