4 ms·

Their AWS bill seems like it would be a lot higher than it needs to be. They're spinning up always-on staging, testing and development environments in 3 differe

Their AWS bill seems like it would be a lot higher than it needs to be. They're spinning up always-on staging, testing and development environments in 3 different regions. I know they've got autoscaling configured, so their production environment should be significantly larger than the others, but it still should be possible to be much more economical while accomplishing all of those non-production workloads. The whole point of infrastructure-as-code should be the ability to spin up environments on-demand, do work, and then spin down. There's no reason to spin up an always-on shared development environment when developers can easily spin up their own environment when they need to do testing and kill it when they're done. Most development tasks can be tested in a single AZ, let alone region. Similarly, QA shouldn't need an always-on, 3-region setup.

blantonl 9y ago
In cloud development environments, you need to account for multi-AZ deployments and the challenges associated with such.
- curun1r 9y ago
  Sure, but in every environment? If a developer is pushing a feature that's mostly business logic, it's probably okay to test in a single-AZ environment. Given their setup, it looks like they still have QA and a final go-no-go test in staging. Issues from a multi-AZ or multi-region setup, should be infrequent enough that you don't need to catch them at the development stage. I'd wager that they'd have far more issues from sharing a development environment between developers who are all pushing unreleased code to the same environment than they'd have from differences with the production environment. It should be pretty easy for a developer to spin up a simplified environment on demand from a branch or specific commit to do their testing. As a bonus, anyone reviewing code can spin up a similar environment from a pull request to ensure that code does what they think it does. Infrastructure as code is a good first step. Embracing the freedom you get from that reproduceability is the next step.
roustem 9y ago
You are right. Our AWS bill could certainly be lower and we will have to start optimizing it at some point. It is does require extra time/effort though. We deployed 1password.ca and 1password.eu just a few months ago and never really got to that. The only thing we "optimize" at the moment is the smaller number and size of EC2 instances in non-production environments.
- curun1r 9y ago
  FWIW, I wasn't trying to be critical. Or, at least, I was trying to be constructively critical. Getting to a Terraform setup is an excellent start...lots of companies can't get that far. I guess I'd just encourage you to view it as the beginning of a longer process that allows you to take full advantage of the fact that AWS bills by the hour and lets you launch as much infrastructure as you want. Always-on non-production environments are, to my mind, a vestige of the time where you had physical servers that needed to be provisioned by a person and cost roughly the same amount when switched off. Or at least the time where OPS built and maintained each AWS VM by hand. On-demand just offers so much more flexibility...stuff like the example from my response to a sibling comment--you should be able to type a single command and spin up an environment from a pull request any time a code reviewer wants to do testing to ensure that what code looks like it does is what it actually does. The more you can leverage the work you've done to get to where you are, the more you can drive down your AWS costs while giving greater flexibility and isolation to your non-prod workload.

blantonl 9y ago

In cloud development environments, you need to account for multi-AZ deployments and the challenges associated with such.

curun1r 9y ago
Sure, but in every environment? If a developer is pushing a feature that's mostly business logic, it's probably okay to test in a single-AZ environment. Given their setup, it looks like they still have QA and a final go-no-go test in staging. Issues from a multi-AZ or multi-region setup, should be infrequent enough that you don't need to catch them at the development stage. I'd wager that they'd have far more issues from sharing a development environment between developers who are all pushing unreleased code to the same environment than they'd have from differences with the production environment. It should be pretty easy for a developer to spin up a simplified environment on demand from a branch or specific commit to do their testing. As a bonus, anyone reviewing code can spin up a similar environment from a pull request to ensure that code does what they think it does. Infrastructure as code is a good first step. Embracing the freedom you get from that reproduceability is the next step.

roustem 9y ago

You are right. Our AWS bill could certainly be lower and we will have to start optimizing it at some point. It is does require extra time/effort though. We deployed 1password.ca and 1password.eu just a few months ago and never really got to that. The only thing we "optimize" at the moment is the smaller number and size of EC2 instances in non-production environments.

curun1r 9y ago
FWIW, I wasn't trying to be critical. Or, at least, I was trying to be constructively critical. Getting to a Terraform setup is an excellent start...lots of companies can't get that far. I guess I'd just encourage you to view it as the beginning of a longer process that allows you to take full advantage of the fact that AWS bills by the hour and lets you launch as much infrastructure as you want. Always-on non-production environments are, to my mind, a vestige of the time where you had physical servers that needed to be provisioned by a person and cost roughly the same amount when switched off. Or at least the time where OPS built and maintained each AWS VM by hand. On-demand just offers so much more flexibility...stuff like the example from my response to a sibling comment--you should be able to type a single command and spin up an environment from a pull request any time a code reviewer wants to do testing to ensure that what code looks like it does is what it actually does. The more you can leverage the work you've done to get to where you are, the more you can drive down your AWS costs while giving greater flexibility and isolation to your non-prod workload.