12 ms·

Terraforming 1Password

346 points by kawera 9y ago

chatmasta 9y ago
I’ve always wondered — from a security perspective, is this kind of an in-depth engineering blogpost a good idea? You’re basically handing a map of your internal infrastructure to any potential attacker who reads the blog. Of course obscurity is not security blah blah blah. Still I can’t help feeling that writeups like this could backfire down the line.
- deweller 9y ago
  On the other hand, you have a team of X employees who work at the company. I'm sure many of them have access to this map. By sharing this with the world, you are encouraged to face any vulnerabilities that you may have overlooked.
- moltar 9y ago
  I imagine the examples are probably fictional. The only thing you know is that they are using AWS.
  - roustem 9y ago
    No, these are real code snippets. I only masked the account identifiers.
- Drakim 9y ago
  On the flipside, being this open might mean that weaknesses come to light because of the increased scrutiny. It's just like when talking about secure communication, you explain exactly how the public/private key exchange works, what algorithms are used, and how the entire handshake takes place. You don't say "We are keeping those details secret just in case it might aid some hacker". The added benefit of everybody examining the process and agreeing that it's sound and secure is just too great to give up.
- geerlingguy 9y ago
  The map / description in the blog post probably describes about 50% of all the applications running in AWS that I’ve seen, almost identically. If they listed out security group and IAM configurations, or how exactly they’re connecting through the bastion, then it would be a little more risky, yes.
  - Jdam 9y ago
    CDN in front of S3, LB in front of Multi-Az EC2 and RDS in background plus Redis for caching... this is basically the infrastructure Hello World of an AWS application. From a security perspective, that might be new for script kiddies but probably any random hacker will assume such a setup.
- vr46 9y ago
  The internal infrastructure of AWS apps are fairly guessable anyway. There are only a limited number of AWS resources and they fit together in predictable ways, VPC - Gateway - ASG - EC2 etc etc. I think it's really great that they've talked about this, it's quite rare to hear about these kinds of internal migrations, and it's something I do a lot with clients but it's not really glamorous enough to talk about.
  - KurtMueller 9y ago
    Hi vr46, where should I go / what resources should I consult if I want to start learning more about AWS & setting up infrastructure? Thanks!
    - alexbilbie 9y ago
      Forgive the self promotion but I'm building a subscription video site - http://stackleap.com/ http://stackleap.com/ - to teach people how to build and deploy to AWS infrastructure. The difference between StackLeap and sites like acloud.guru is that I'm focussing on the day to day stuff rather than high level concepts you need to know to pass the certification exams. I'm hoping to launch a beta in the next few weeks.
      - insomniacity 9y ago
        I've been very impressed with CloudAcademy - it seems to issue AWS accounts per course/per lab - it guides you through the steps and check whether you completed them by checking the AWS account, and then tears it down at the end. You might not be ready for that level of integration, but see if you can get a demo or trial.
    - vr46 9y ago
      Hi KurtMueller, get yourself a new AWS account with the free tier, find out exactly which resources ARE NOT included with the free tier, and get down with a simple Terraform file. Terraform code has a one-to-one mapping with AWS resources, so it's easy to follow. One code test that my previous company had was to create a simple load-balanced web server in Ansible or Chef, but you could also do it with Terraform too. Two servers and a load balancer in front, simple but teaches you a lot of fundamental stuff.
- roustem 9y ago
  If our security depended on this information being private then we would be in big trouble. I would love to go into more details about the environment and some of the things we did to build on top of the default AWS security settings. It was too much information for this post, maybe we will do another one that focuses on security.
- brown9-2 9y ago
  All this tells me is that they use EC2, Aurora, and other AWS products. What am I supposed to do with that?
geerlingguy 9y ago
The code comparison between CloudFormation’s abysmal JSON formatting and Terraform’s DSL is a bit disingenuous. CloudFormation has supported YAML for at least a year or two now, and it’s leagues more readable and compact, not to mention maintainable—you can even add comments to your code with YAML (something that is impossible with the old JSON format). I’ve spent a lot of time working between the two, and while Terraform does have a lot to offer and is often a very valid option, CloudFormation has its virtues, especially the fact that almost all bleeding edge AWS features are first available to be managed via CloudFormation (sometimes with weeks or months of lead time), and many bugs can be more readily ironed out with AWS support (assuming you have it). Again, not saying don’t use Terraform, just that I don’t think the decision is quite as black and white as this blog post seems to make it.
- Thaxll 9y ago
  CloudFormation is really bad compared to TF.
  - oblio 9y ago
    Because...?
- djhworld 9y ago
  Personally I find YAML for cloudformation worse than JSON because of the whitespacing requirements. JSON isn't much better mind, especially once your template reaches to hundreds of lines with nested objects. Recently I've come around to using Troposphere [1] to write cloudformation templates, it's actually very pleasant to use. You just write your infrastructure in python, and it will generate you a template at the end. The developers seem to respond quickly to changes AWS make to the CF templating language too. [1] https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
  - GhostVII 9y ago
    JSON is a subset of yaml, so if you want to just use JSON it should still work.
merb 9y ago
would've been easier to just migrate to k8s and used some kind of ignition/managed k8s (and maybe ansible, if things still needs some manual tweaking).
- kuschku 9y ago
  Let's make one thing very clear here - k8s is never easy. I've been running a cluster since it became usable, and it definitely is the opposite of easy. Kubernetes is powerful, it is modular, and it makes everything a lot more efficient, but setting it up - especially in such large deployments as 1Password would have here - is never easy.
  - _ix 9y ago
    Any recommendations toward primers and getting started with k8s?
    - kuschku 9y ago
      Manning's Kubernetes in action is a good start, additionally I can recommend everything Kelsey Hightower created on that topic. Also, I recommend that you first try using kubernetes before you dive in with setting it up. Kubernetes.io has a live tutorial where you can work with a minikube cluster in your browser, afterwards you might want to use Google Cloud's free tier for a small Kubernetes cluster or Minikube until you're comfortable with kubernetes. For seting up your own cluster, I've heard great stuff about kubernetes the hard way – not to actually set one up, but to learn how the internals work, so you can then fix issues in the cluster you’ll set up with kubeadm/kops/GKE
- roustem 9y ago
  I would love to migrate to k8s at some point. It would make some things easier. We still need to figure out how to do that without downgrading our existing security configuration -- a lot of it depends on running different services in their own subnets and AWS security roles. It might be easier now that AWS is starting to support Kubernetes.
- 9y ago
mterwill 9y ago
Interesting blog post, but AgileBits should have communicated planned downtime to their customers via email, which they did not. I’ve been a 1Password user for years and recently switched to their hosted offering. I know a massive infrastructure migration is rare, but that’s all the more reason to be transparent ahead of time.
- blantonl 9y ago
  What impact did this changeover actually have on you?
- roustem 9y ago
  I am sorry if the downtime affected you in any way. Sending several million emails could be a challenge considering that most of our customers depend on 1Password apps and usually not affected by the downtime. We do have a status page and Twitter feed where we make announcements: https://status.1password.com https://status.1password.com https://twitter.com/1passwordstatus https://twitter.com/1passwordstatus
  - NelsonMinar 9y ago
    Have you considered building a notification system into the app?
fredsted 9y ago
If you want to try out Terraform yourself for a simple Web server/DB Server/Load balancer infrastructure, I made a tutorial for just that here: https://simonfredsted.com/1459 https://simonfredsted.com/1459
curun1r 9y ago
Their AWS bill seems like it would be a lot higher than it needs to be. They're spinning up always-on staging, testing and development environments in 3 different regions. I know they've got autoscaling configured, so their production environment should be significantly larger than the others, but it still should be possible to be much more economical while accomplishing all of those non-production workloads. The whole point of infrastructure-as-code should be the ability to spin up environments on-demand, do work, and then spin down. There's no reason to spin up an always-on shared development environment when developers can easily spin up their own environment when they need to do testing and kill it when they're done. Most development tasks can be tested in a single AZ, let alone region. Similarly, QA shouldn't need an always-on, 3-region setup.
- blantonl 9y ago
  In cloud development environments, you need to account for multi-AZ deployments and the challenges associated with such.
  - curun1r 9y ago
    Sure, but in every environment? If a developer is pushing a feature that's mostly business logic, it's probably okay to test in a single-AZ environment. Given their setup, it looks like they still have QA and a final go-no-go test in staging. Issues from a multi-AZ or multi-region setup, should be infrequent enough that you don't need to catch them at the development stage. I'd wager that they'd have far more issues from sharing a development environment between developers who are all pushing unreleased code to the same environment than they'd have from differences with the production environment. It should be pretty easy for a developer to spin up a simplified environment on demand from a branch or specific commit to do their testing. As a bonus, anyone reviewing code can spin up a similar environment from a pull request to ensure that code does what they think it does. Infrastructure as code is a good first step. Embracing the freedom you get from that reproduceability is the next step.
- roustem 9y ago
  You are right. Our AWS bill could certainly be lower and we will have to start optimizing it at some point. It is does require extra time/effort though. We deployed 1password.ca and 1password.eu just a few months ago and never really got to that. The only thing we "optimize" at the moment is the smaller number and size of EC2 instances in non-production environments.
rgoodwintx 9y ago
Side note, for anyone wondering like I was, the visualization tool is Cloudcraft: https://cloudcraft.co/ https://cloudcraft.co/ . I'd love to have something similar for doing isometric views of any kind of diagramming.
- casey_lang 9y ago
  Every time I see Cloudcraft, I wish it was available for GCP. I think it's a fantastic way to visualize interconnected systems
  - user5994461 9y ago
    Ah. I wondered what made these terrible diagrams. They look cute but the text and the symbols are incomprehensible.
    - jchw 9y ago
      Looks mostly comprehensible to me. A lot of the icons are obvious if you deal with this stuff a lot, like the ELB, Redis icons, and the ones with instance types written on them. Bit too zoomed out to read the small text, though.
      - philwelch 9y ago
        If they didn't have the precious isometric projection and just showed a regular 2D diagram it would be more legible.
- learn_more 9y ago
  Have a look at https://www.pathwaysystems.com/video/#racks1 https://www.pathwaysystems.com/video/#racks1 The video is a bit slow because it's meant to be instructional. There are other videos on the same page which show how the model (graph) can be viewed logically instead of isometrically (what we call spatially). I'm the CTO, happy to answer questions.
mcrmonkey 9y ago
What are your thoughts on using modules inside terraform ?
- _ix 9y ago
  Not the op, but I have some brief thoughts. Given a bit of time with Terraform, the need for modules becomes obvious as you identify common resources for the infrastructure you're modeling. I thought writing modules first was the "right way," but starting with modules ended up being a waste.
  - scaryclam 9y ago
    I'll second this. I started out not using modules, but figured I'd migrate things over to using them as and when it would make things tidier. I never migrated things. Modules are really neat but I don't think they're a sensible starting point if you're not doing a lot of repetition or don't exactly know what you need to build yet.
- stimur 9y ago
  Everything described in the blog post is using modules making infrastructure description abstract enough to understand and deploy/manage.
jbergknoff 9y ago
Great post. It's always good to see more examples of people putting these tools to work. With that said, I consider getting the AMI id dynamically to be an anti-pattern which undermines the principles of infrastructure-as-code. Specifically, it introduces an implicit build variable "time of `terraform apply`", which is not tracked in version control. Happily, because of Terraform's design, this sort of thing mostly won't cause your infrastructure to drift into unexpected states (e.g. production instance 1 running AMI X and production instance 2 running AMI Y). Within an environment, things should be consistent, but your staging environment may run AMI X while production is running AMI Y, and you wouldn't know from looking at your Terraform definitions. I previously wrote about similar ideas in the context of pinning dependency versions, where wildcards can and often do get you into bad states. https://jonathan.bergknoff.com/journal/always-pin-your-versions https://jonathan.bergknoff.com/journal/always-pin-your-versi...
- _ix 9y ago
  This is a really important point. Our company is using Terraform in a limited way, but Puppet is our primary automated configuration management tool. Similarly in Puppet, `ensure => latest,` on package resources isn't necessarily dangerous, but it can add a lot of confusion without intentional commits against the control repositories.
  - jo909 9y ago
    Puppet rules are typically applied constantly in short intervals. The installed version should always be the same everywhere, the latest available from the repository. The risk is more that you can get updates installed at inopportune times. Of course what's right is that the version change is not reflected in the configuration management, but this isn't normally a problem for minor version changes.
    - _ix 9y ago
      I think the default settings for puppet agents are typically 30 minutes if I'm not mistaken. I've inherited a bit of a DevOps mess with some 600-700 nodes in various states of management. In previous positions, there was a great hew and cry that the run intervals were increased from 30 minutes to 60 minutes... eventually every four hours for production resources. In my current position, production nodes are provisioned to run puppet once daily as a rule, triggered by cron jobs at a pseudorandom minute between 0200 and 0400 defined at server provisioning time. How do these intervals compare to what you've seen?
notyourday 9y ago
Wait. People are discussing and giving accolades how marvelous it is that a company is migrating infrastructure to a favor of the month because it is using a flavor of some other month tools when the company has the audacity to say they would be down for hours? Are you kidding?
- _ix 9y ago
  I don't think there is any shortage of literature on the need to avoid vendor lock-in. Selecting a cloud agnostic tool like those developed by Hashicorp and the opensource community offers the folks at 1Password additional flexibility in their choice of cloud providers. AWS is great, but still... I don't see any mention of Terraform Enterprise here, either. I imagine they're perfectly capable of pursuing Terraform with the foss version, although the enterprise complement has some pretty great additional features.
  - notyourday 9y ago
    It is possible I was not clear: The blog post is patting itself on a back for migration that caused downtime. It included this gem: > Couldn’t you’ve imported all online resources? Just > wondering. > That is certainly possible, and it would have allowed us > to avoid downtime. Unfortunately, it also > requires manual mapping of all existing resources. > Because of that, it’s hard to test, and the chance of a > human error is high – and we know humans are pretty bad > at this. As a wise person on Twitter said: “If you can’t > rebuild it, you can’t rebuild it“.
wgjordan 9y ago
I use CloudFormation to manage similar AWS web-app infrastructure. I've been continuously evaluating Terraform over the years (it is indeed maturing quickly), but have still decided to stick with CloudFormation for now, and would still continue to recommend the same for anyone managing an AWS-exclusive (or mostly-AWS) deployment. To respond to some specific items mentioned in this post: - "Terraform has a more straightforward and powerful language (HCL) that makes it easier to write and review code." It's easy to pipe configuration through your favorite 'straightforward and powerful' templating language of choice to generate the stack template used by CloudFormation. I definitely wouldn't use CloudFormation at all without an extra preprocessing step of some sort. - "Terraform has another gem of a feature that we rely on: terraform plan. It allows us to visualize the changes that will happen to the environment without performing them." CloudFormation has a similar feature called "Change Sets" (released in March 2016). Finally, two more things to note: - CloudFormation supports "Custom Resources", which allow you to write Lambda-function scripts to perform any custom operation you want (e.g., interact with third-party APIs, or support new AWS resources that don't yet have an official CloudFormation resource implementation). There's not as extensive a library of providers Terraform supports, but you can often find open-source providers to fill the gaps on AWS resource until an official implementation is released. And if not, you can just write a quick and dirty implementation yourself. Terraform supports a similar feature in "Custom Providers", but they are Go-only as opposed to the various supports Lambda-language runtimes. - CloudFormation and all official AWS-resource implementations not only have excellent public documentation but are officially supported by AWS Support, which is a big deal if you're deploying something with any degree of complexity or cost/risk. As for Terraform's strengths: its extensive set of resource providers beyond AWS ecosystem can't be beat. Also, it has great support for mapping existing infrastructure resources, which is a feature still sorely lacking in CloudFormation. (I'm surprised the author of this post didn't take advantage of this for their migration to avoid downtime!)
- ben0x539 9y ago
  > It's easy to pipe configuration through your favorite 'straightforward and powerful' templating language of choice to generate the stack template used by CloudFormation. I definitely wouldn't use CloudFormation at all without an extra preprocessing step of some sort. The tricky bit is what kind of logic you can apply to values that are the result of provisioning/looking up resources in the first place, isn't it? Like, what dynamically determined resource IDs are exposed to the language and in what places you can reference them. Basically the capability of terraform's interpolation language vs. CloudFormation Fn::* constructs, I suppose.
philsnow 9y ago
Here I am wishing that CloudFormation had one killer feature that would have allowed us to use it at work: the ability to adopt existing resources into a CF stack. When we were starting on the path of "hey maybe all our infra shouldn't be pointy clicky", we chose between CloudFormation, Terraform, and making something in-house. Out of those three, Terraform was the clear winner for us at the time, but it has not been without issues. Nobody else in this whole thread seems to be complaining about state management. I think it's insane that Terraform encodes where a given resource is in your filesystem / module hierarchy into the state JSON structure. (Unless something has changed since I last looked,) if you want to move things around in your .tf source, terraform can only apply that by tearing down the old resource and recreating it. For our large setup, in order to adopt Terraform, we've had to spend a ton of time upfront thinking very hard about how all of our .tf sources are going to look, and it's delayed our deployment by months.
- steveh73 9y ago
  You can move state around with `terraform state mv`. It's a bit tricky and has some gotchas but works (which applies to terraform as a whole really)
- raziel2p 9y ago
  Terraform allows you to make modifications to the state file yourself, both with terraform commands like terraform state mv, or manually if you're brave enough to edit the JSON. It requires confidence in using the tool, of course. But it also encourages you to create your cloud resources in a way where it's safe to let Terraform destroy and re-create most of it at any time.
  - bchallenor 9y ago
    `terraform state mv` is indeed the trick. It took me a while to understand it, but this blog post helped. [1] It leads you through refactoring some resources into a module. The key takeaway for me was "we really only need to consider the nodes that map to the physical resources of our infrastructure when we are planning our state surgery. This means we can ignore all of the nodes that correspond to data sources, variables, and providers." So after a refactor, this is what I do now: (1) run plan to get the names of everything terraform wants to delete and recreate; (2) pair all the resource nodes manually and translate them to state mv commands; (3) re-run plan and verify that terraform is now convinced there is nothing to do. It would be nice if terraform could do this for me, of course, but I find that it is generally possible to avoid delete and recreate if all I've done is a refactoring. [1] https://ryaneschinger.com/blog/terraform-state-move/ https://ryaneschinger.com/blog/terraform-state-move/
alexnewman 9y ago
Two of my least favorite technology. I have been using pass for a while and will never go back
GordonS 9y ago
Wow, I found it fascinating to read such a detailed description of the architecture of a running business! Is anyone else aware of similar blog posts from other companies?
- Zaheer 9y ago
  I've found http://highscalability.com/ http://highscalability.com/ & https://stackshare.io/featured-posts https://stackshare.io/featured-posts to be really useful. I constantly refer to these at work for design inspiration.
acd 9y ago
There is a Terraforming github project which exports existing Amazon EC2 infrastructure and creates Terraform infrastructure tf code and tfstate. Terraforming github repo does not seem to be related to 1password. Terraforming github repo https://github.com/dtan4/terraforming https://github.com/dtan4/terraforming http://terraforming.dtan4.net/ http://terraforming.dtan4.net/
somtum 9y ago
One major drawback of Cloudformation is its inability to view changesets of nested templates.
somtum 9y ago
With infrastructure in multiple regions, do you have independent Aurora databases running in each region? Do you have any data synchronising between regions or are they running completely independently?
- stimur 9y ago
  They are completely independent, that was the goal.
dev2ops 9y ago
Awesome post! Always happy to see more committed infrastructure. We made this move a while back and were really happy with the results. Check out Packer for AMI builds and you're all set. One suggestion re: TF files is to keep each service in a separate TF file/state and keep all your state files on S3 that way if someone does an apply it's always consistent. Keeping things separated means that you don't have to worry about hitting other services when you do an apply
sridca 9y ago
Terraform manages infrastructure; if you also want to manage the installed software in a declarative and reproducible way consider NixOps (based on Nix/ NixOS): http://container-solutions.com/step-towards-future-configuration-infrastructure-management-nix/ http://container-solutions.com/step-towards-future-configura...
juliosueiras 9y ago
If you are using vim for terraform, then you should try this https://github.com/juliosueiras/vim-terraform-completion https://github.com/juliosueiras/vim-terraform-completion, hoping to migrate to langserver by this year so any editor can use it P.S. sorry for the self-promo
gergnz 9y ago
I find that both terraform and cloudformation have their pros and cons like many things in life. It comes down to personal preference and how you think (vim/Emacs?). For me, I choose cfndsl to build my cloudformation templates. This gives me the ability to actually write code. JSON, YAML, HCL aren't full languages. Yes tf and cf try to provide language like concepts (iteration, etc) but really at the end of the day they are just a definition of your environment and not strictly code. Note: I maintain cfndsl, so I'm totally bias.
paulddraper 9y ago
> servers will be down for the next few hours. We are recreating our entire environment to replace AWS CloudFormation with @HashiCorp Terraform. One of the greatest strengths of terraform (vs say, cloudformation) is that you can adopt existing resources, and ZERO downtime is needed to migrate. Why take downtime?
- stimur 9y ago
  Because we needed to re-deploy our database, which unfortunately requires downtime.

346 points by kawera 9y ago

I’ve always wondered — from a security perspective, is this kind of an in-depth engineering blogpost a good idea? You’re basically handing a map of your internal infrastructure to any potential attacker who reads the blog. Of course obscurity is not security blah blah blah. Still I can’t help feeling that writeups like this could backfire down the line.

deweller 9y ago
On the other hand, you have a team of X employees who work at the company. I'm sure many of them have access to this map. By sharing this with the world, you are encouraged to face any vulnerabilities that you may have overlooked.
moltar 9y ago
I imagine the examples are probably fictional. The only thing you know is that they are using AWS.
- roustem 9y ago
  No, these are real code snippets. I only masked the account identifiers.
Drakim 9y ago
On the flipside, being this open might mean that weaknesses come to light because of the increased scrutiny. It's just like when talking about secure communication, you explain exactly how the public/private key exchange works, what algorithms are used, and how the entire handshake takes place. You don't say "We are keeping those details secret just in case it might aid some hacker". The added benefit of everybody examining the process and agreeing that it's sound and secure is just too great to give up.
geerlingguy 9y ago
The map / description in the blog post probably describes about 50% of all the applications running in AWS that I’ve seen, almost identically. If they listed out security group and IAM configurations, or how exactly they’re connecting through the bastion, then it would be a little more risky, yes.
- Jdam 9y ago
  CDN in front of S3, LB in front of Multi-Az EC2 and RDS in background plus Redis for caching... this is basically the infrastructure Hello World of an AWS application. From a security perspective, that might be new for script kiddies but probably any random hacker will assume such a setup.
vr46 9y ago
The internal infrastructure of AWS apps are fairly guessable anyway. There are only a limited number of AWS resources and they fit together in predictable ways, VPC - Gateway - ASG - EC2 etc etc. I think it's really great that they've talked about this, it's quite rare to hear about these kinds of internal migrations, and it's something I do a lot with clients but it's not really glamorous enough to talk about.
- KurtMueller 9y ago
  Hi vr46, where should I go / what resources should I consult if I want to start learning more about AWS & setting up infrastructure? Thanks!
  - alexbilbie 9y ago
    Forgive the self promotion but I'm building a subscription video site - http://stackleap.com/ http://stackleap.com/ - to teach people how to build and deploy to AWS infrastructure. The difference between StackLeap and sites like acloud.guru is that I'm focussing on the day to day stuff rather than high level concepts you need to know to pass the certification exams. I'm hoping to launch a beta in the next few weeks.
    - insomniacity 9y ago
      I've been very impressed with CloudAcademy - it seems to issue AWS accounts per course/per lab - it guides you through the steps and check whether you completed them by checking the AWS account, and then tears it down at the end. You might not be ready for that level of integration, but see if you can get a demo or trial.
  - vr46 9y ago
    Hi KurtMueller, get yourself a new AWS account with the free tier, find out exactly which resources ARE NOT included with the free tier, and get down with a simple Terraform file. Terraform code has a one-to-one mapping with AWS resources, so it's easy to follow. One code test that my previous company had was to create a simple load-balanced web server in Ansible or Chef, but you could also do it with Terraform too. Two servers and a load balancer in front, simple but teaches you a lot of fundamental stuff.
roustem 9y ago
If our security depended on this information being private then we would be in big trouble. I would love to go into more details about the environment and some of the things we did to build on top of the default AWS security settings. It was too much information for this post, maybe we will do another one that focuses on security.
brown9-2 9y ago
All this tells me is that they use EC2, Aurora, and other AWS products. What am I supposed to do with that?

geerlingguy 9y ago

The code comparison between CloudFormation’s abysmal JSON formatting and Terraform’s DSL is a bit disingenuous. CloudFormation has supported YAML for at least a year or two now, and it’s leagues more readable and compact, not to mention maintainable—you can even add comments to your code with YAML (something that is impossible with the old JSON format). I’ve spent a lot of time working between the two, and while Terraform does have a lot to offer and is often a very valid option, CloudFormation has its virtues, especially the fact that almost all bleeding edge AWS features are first available to be managed via CloudFormation (sometimes with weeks or months of lead time), and many bugs can be more readily ironed out with AWS support (assuming you have it). Again, not saying don’t use Terraform, just that I don’t think the decision is quite as black and white as this blog post seems to make it.

Thaxll 9y ago
CloudFormation is really bad compared to TF.
- oblio 9y ago
  Because...?
djhworld 9y ago
Personally I find YAML for cloudformation worse than JSON because of the whitespacing requirements. JSON isn't much better mind, especially once your template reaches to hundreds of lines with nested objects. Recently I've come around to using Troposphere [1] to write cloudformation templates, it's actually very pleasant to use. You just write your infrastructure in python, and it will generate you a template at the end. The developers seem to respond quickly to changes AWS make to the CF templating language too. [1] https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
- GhostVII 9y ago
  JSON is a subset of yaml, so if you want to just use JSON it should still work.

merb 9y ago

would've been easier to just migrate to k8s and used some kind of ignition/managed k8s (and maybe ansible, if things still needs some manual tweaking).

kuschku 9y ago
Let's make one thing very clear here - k8s is never easy. I've been running a cluster since it became usable, and it definitely is the opposite of easy. Kubernetes is powerful, it is modular, and it makes everything a lot more efficient, but setting it up - especially in such large deployments as 1Password would have here - is never easy.
- _ix 9y ago
  Any recommendations toward primers and getting started with k8s?
  - kuschku 9y ago
    Manning's Kubernetes in action is a good start, additionally I can recommend everything Kelsey Hightower created on that topic. Also, I recommend that you first try using kubernetes before you dive in with setting it up. Kubernetes.io has a live tutorial where you can work with a minikube cluster in your browser, afterwards you might want to use Google Cloud's free tier for a small Kubernetes cluster or Minikube until you're comfortable with kubernetes. For seting up your own cluster, I've heard great stuff about kubernetes the hard way – not to actually set one up, but to learn how the internals work, so you can then fix issues in the cluster you’ll set up with kubeadm/kops/GKE
roustem 9y ago
I would love to migrate to k8s at some point. It would make some things easier. We still need to figure out how to do that without downgrading our existing security configuration -- a lot of it depends on running different services in their own subnets and AWS security roles. It might be easier now that AWS is starting to support Kubernetes.
9y ago

mterwill 9y ago

Interesting blog post, but AgileBits should have communicated planned downtime to their customers via email, which they did not. I’ve been a 1Password user for years and recently switched to their hosted offering. I know a massive infrastructure migration is rare, but that’s all the more reason to be transparent ahead of time.

blantonl 9y ago
What impact did this changeover actually have on you?
roustem 9y ago
I am sorry if the downtime affected you in any way. Sending several million emails could be a challenge considering that most of our customers depend on 1Password apps and usually not affected by the downtime. We do have a status page and Twitter feed where we make announcements: https://status.1password.com https://status.1password.com https://twitter.com/1passwordstatus https://twitter.com/1passwordstatus
- NelsonMinar 9y ago
  Have you considered building a notification system into the app?

fredsted 9y ago

If you want to try out Terraform yourself for a simple Web server/DB Server/Load balancer infrastructure, I made a tutorial for just that here: https://simonfredsted.com/1459 https://simonfredsted.com/1459

curun1r 9y ago

Their AWS bill seems like it would be a lot higher than it needs to be. They're spinning up always-on staging, testing and development environments in 3 different regions. I know they've got autoscaling configured, so their production environment should be significantly larger than the others, but it still should be possible to be much more economical while accomplishing all of those non-production workloads. The whole point of infrastructure-as-code should be the ability to spin up environments on-demand, do work, and then spin down. There's no reason to spin up an always-on shared development environment when developers can easily spin up their own environment when they need to do testing and kill it when they're done. Most development tasks can be tested in a single AZ, let alone region. Similarly, QA shouldn't need an always-on, 3-region setup.

blantonl 9y ago
In cloud development environments, you need to account for multi-AZ deployments and the challenges associated with such.
- curun1r 9y ago
  Sure, but in every environment? If a developer is pushing a feature that's mostly business logic, it's probably okay to test in a single-AZ environment. Given their setup, it looks like they still have QA and a final go-no-go test in staging. Issues from a multi-AZ or multi-region setup, should be infrequent enough that you don't need to catch them at the development stage. I'd wager that they'd have far more issues from sharing a development environment between developers who are all pushing unreleased code to the same environment than they'd have from differences with the production environment. It should be pretty easy for a developer to spin up a simplified environment on demand from a branch or specific commit to do their testing. As a bonus, anyone reviewing code can spin up a similar environment from a pull request to ensure that code does what they think it does. Infrastructure as code is a good first step. Embracing the freedom you get from that reproduceability is the next step.
roustem 9y ago
You are right. Our AWS bill could certainly be lower and we will have to start optimizing it at some point. It is does require extra time/effort though. We deployed 1password.ca and 1password.eu just a few months ago and never really got to that. The only thing we "optimize" at the moment is the smaller number and size of EC2 instances in non-production environments.

rgoodwintx 9y ago

Side note, for anyone wondering like I was, the visualization tool is Cloudcraft: https://cloudcraft.co/ https://cloudcraft.co/ . I'd love to have something similar for doing isometric views of any kind of diagramming.

casey_lang 9y ago
Every time I see Cloudcraft, I wish it was available for GCP. I think it's a fantastic way to visualize interconnected systems
- user5994461 9y ago
  Ah. I wondered what made these terrible diagrams. They look cute but the text and the symbols are incomprehensible.
  - jchw 9y ago
    Looks mostly comprehensible to me. A lot of the icons are obvious if you deal with this stuff a lot, like the ELB, Redis icons, and the ones with instance types written on them. Bit too zoomed out to read the small text, though.
    - philwelch 9y ago
      If they didn't have the precious isometric projection and just showed a regular 2D diagram it would be more legible.
learn_more 9y ago
Have a look at https://www.pathwaysystems.com/video/#racks1 https://www.pathwaysystems.com/video/#racks1 The video is a bit slow because it's meant to be instructional. There are other videos on the same page which show how the model (graph) can be viewed logically instead of isometrically (what we call spatially). I'm the CTO, happy to answer questions.

mcrmonkey 9y ago

What are your thoughts on using modules inside terraform ?

_ix 9y ago
Not the op, but I have some brief thoughts. Given a bit of time with Terraform, the need for modules becomes obvious as you identify common resources for the infrastructure you're modeling. I thought writing modules first was the "right way," but starting with modules ended up being a waste.
- scaryclam 9y ago
  I'll second this. I started out not using modules, but figured I'd migrate things over to using them as and when it would make things tidier. I never migrated things. Modules are really neat but I don't think they're a sensible starting point if you're not doing a lot of repetition or don't exactly know what you need to build yet.
stimur 9y ago
Everything described in the blog post is using modules making infrastructure description abstract enough to understand and deploy/manage.

jbergknoff 9y ago

Great post. It's always good to see more examples of people putting these tools to work. With that said, I consider getting the AMI id dynamically to be an anti-pattern which undermines the principles of infrastructure-as-code. Specifically, it introduces an implicit build variable "time of `terraform apply`", which is not tracked in version control. Happily, because of Terraform's design, this sort of thing mostly won't cause your infrastructure to drift into unexpected states (e.g. production instance 1 running AMI X and production instance 2 running AMI Y). Within an environment, things should be consistent, but your staging environment may run AMI X while production is running AMI Y, and you wouldn't know from looking at your Terraform definitions. I previously wrote about similar ideas in the context of pinning dependency versions, where wildcards can and often do get you into bad states. https://jonathan.bergknoff.com/journal/always-pin-your-versions https://jonathan.bergknoff.com/journal/always-pin-your-versi...

_ix 9y ago
This is a really important point. Our company is using Terraform in a limited way, but Puppet is our primary automated configuration management tool. Similarly in Puppet, `ensure => latest,` on package resources isn't necessarily dangerous, but it can add a lot of confusion without intentional commits against the control repositories.
- jo909 9y ago
  Puppet rules are typically applied constantly in short intervals. The installed version should always be the same everywhere, the latest available from the repository. The risk is more that you can get updates installed at inopportune times. Of course what's right is that the version change is not reflected in the configuration management, but this isn't normally a problem for minor version changes.
  - _ix 9y ago
    I think the default settings for puppet agents are typically 30 minutes if I'm not mistaken. I've inherited a bit of a DevOps mess with some 600-700 nodes in various states of management. In previous positions, there was a great hew and cry that the run intervals were increased from 30 minutes to 60 minutes... eventually every four hours for production resources. In my current position, production nodes are provisioned to run puppet once daily as a rule, triggered by cron jobs at a pseudorandom minute between 0200 and 0400 defined at server provisioning time. How do these intervals compare to what you've seen?

notyourday 9y ago

Wait. People are discussing and giving accolades how marvelous it is that a company is migrating infrastructure to a favor of the month because it is using a flavor of some other month tools when the company has the audacity to say they would be down for hours? Are you kidding?

_ix 9y ago
I don't think there is any shortage of literature on the need to avoid vendor lock-in. Selecting a cloud agnostic tool like those developed by Hashicorp and the opensource community offers the folks at 1Password additional flexibility in their choice of cloud providers. AWS is great, but still... I don't see any mention of Terraform Enterprise here, either. I imagine they're perfectly capable of pursuing Terraform with the foss version, although the enterprise complement has some pretty great additional features.
- notyourday 9y ago
  It is possible I was not clear: The blog post is patting itself on a back for migration that caused downtime. It included this gem: > Couldn’t you’ve imported all online resources? Just > wondering. > That is certainly possible, and it would have allowed us > to avoid downtime. Unfortunately, it also > requires manual mapping of all existing resources. > Because of that, it’s hard to test, and the chance of a > human error is high – and we know humans are pretty bad > at this. As a wise person on Twitter said: “If you can’t > rebuild it, you can’t rebuild it“.

wgjordan 9y ago

I use CloudFormation to manage similar AWS web-app infrastructure. I've been continuously evaluating Terraform over the years (it is indeed maturing quickly), but have still decided to stick with CloudFormation for now, and would still continue to recommend the same for anyone managing an AWS-exclusive (or mostly-AWS) deployment. To respond to some specific items mentioned in this post: - "Terraform has a more straightforward and powerful language (HCL) that makes it easier to write and review code." It's easy to pipe configuration through your favorite 'straightforward and powerful' templating language of choice to generate the stack template used by CloudFormation. I definitely wouldn't use CloudFormation at all without an extra preprocessing step of some sort. - "Terraform has another gem of a feature that we rely on: terraform plan. It allows us to visualize the changes that will happen to the environment without performing them." CloudFormation has a similar feature called "Change Sets" (released in March 2016). Finally, two more things to note: - CloudFormation supports "Custom Resources", which allow you to write Lambda-function scripts to perform any custom operation you want (e.g., interact with third-party APIs, or support new AWS resources that don't yet have an official CloudFormation resource implementation). There's not as extensive a library of providers Terraform supports, but you can often find open-source providers to fill the gaps on AWS resource until an official implementation is released. And if not, you can just write a quick and dirty implementation yourself. Terraform supports a similar feature in "Custom Providers", but they are Go-only as opposed to the various supports Lambda-language runtimes. - CloudFormation and all official AWS-resource implementations not only have excellent public documentation but are officially supported by AWS Support, which is a big deal if you're deploying something with any degree of complexity or cost/risk. As for Terraform's strengths: its extensive set of resource providers beyond AWS ecosystem can't be beat. Also, it has great support for mapping existing infrastructure resources, which is a feature still sorely lacking in CloudFormation. (I'm surprised the author of this post didn't take advantage of this for their migration to avoid downtime!)

ben0x539 9y ago
> It's easy to pipe configuration through your favorite 'straightforward and powerful' templating language of choice to generate the stack template used by CloudFormation. I definitely wouldn't use CloudFormation at all without an extra preprocessing step of some sort. The tricky bit is what kind of logic you can apply to values that are the result of provisioning/looking up resources in the first place, isn't it? Like, what dynamically determined resource IDs are exposed to the language and in what places you can reference them. Basically the capability of terraform's interpolation language vs. CloudFormation Fn::* constructs, I suppose.

philsnow 9y ago

Here I am wishing that CloudFormation had one killer feature that would have allowed us to use it at work: the ability to adopt existing resources into a CF stack. When we were starting on the path of "hey maybe all our infra shouldn't be pointy clicky", we chose between CloudFormation, Terraform, and making something in-house. Out of those three, Terraform was the clear winner for us at the time, but it has not been without issues. Nobody else in this whole thread seems to be complaining about state management. I think it's insane that Terraform encodes where a given resource is in your filesystem / module hierarchy into the state JSON structure. (Unless something has changed since I last looked,) if you want to move things around in your .tf source, terraform can only apply that by tearing down the old resource and recreating it. For our large setup, in order to adopt Terraform, we've had to spend a ton of time upfront thinking very hard about how all of our .tf sources are going to look, and it's delayed our deployment by months.

steveh73 9y ago
You can move state around with `terraform state mv`. It's a bit tricky and has some gotchas but works (which applies to terraform as a whole really)
raziel2p 9y ago
Terraform allows you to make modifications to the state file yourself, both with terraform commands like terraform state mv, or manually if you're brave enough to edit the JSON. It requires confidence in using the tool, of course. But it also encourages you to create your cloud resources in a way where it's safe to let Terraform destroy and re-create most of it at any time.
- bchallenor 9y ago
  `terraform state mv` is indeed the trick. It took me a while to understand it, but this blog post helped. [1] It leads you through refactoring some resources into a module. The key takeaway for me was "we really only need to consider the nodes that map to the physical resources of our infrastructure when we are planning our state surgery. This means we can ignore all of the nodes that correspond to data sources, variables, and providers." So after a refactor, this is what I do now: (1) run plan to get the names of everything terraform wants to delete and recreate; (2) pair all the resource nodes manually and translate them to state mv commands; (3) re-run plan and verify that terraform is now convinced there is nothing to do. It would be nice if terraform could do this for me, of course, but I find that it is generally possible to avoid delete and recreate if all I've done is a refactoring. [1] https://ryaneschinger.com/blog/terraform-state-move/ https://ryaneschinger.com/blog/terraform-state-move/

alexnewman 9y ago

Two of my least favorite technology. I have been using pass for a while and will never go back

GordonS 9y ago

Wow, I found it fascinating to read such a detailed description of the architecture of a running business! Is anyone else aware of similar blog posts from other companies?

Zaheer 9y ago
I've found http://highscalability.com/ http://highscalability.com/ & https://stackshare.io/featured-posts https://stackshare.io/featured-posts to be really useful. I constantly refer to these at work for design inspiration.

acd 9y ago

There is a Terraforming github project which exports existing Amazon EC2 infrastructure and creates Terraform infrastructure tf code and tfstate. Terraforming github repo does not seem to be related to 1password. Terraforming github repo https://github.com/dtan4/terraforming https://github.com/dtan4/terraforming http://terraforming.dtan4.net/ http://terraforming.dtan4.net/

somtum 9y ago

One major drawback of Cloudformation is its inability to view changesets of nested templates.

somtum 9y ago

With infrastructure in multiple regions, do you have independent Aurora databases running in each region? Do you have any data synchronising between regions or are they running completely independently?

stimur 9y ago
They are completely independent, that was the goal.

dev2ops 9y ago

Awesome post! Always happy to see more committed infrastructure. We made this move a while back and were really happy with the results. Check out Packer for AMI builds and you're all set. One suggestion re: TF files is to keep each service in a separate TF file/state and keep all your state files on S3 that way if someone does an apply it's always consistent. Keeping things separated means that you don't have to worry about hitting other services when you do an apply

sridca 9y ago

Terraform manages infrastructure; if you also want to manage the installed software in a declarative and reproducible way consider NixOps (based on Nix/ NixOS): http://container-solutions.com/step-towards-future-configuration-infrastructure-management-nix/ http://container-solutions.com/step-towards-future-configura...

juliosueiras 9y ago

If you are using vim for terraform, then you should try this https://github.com/juliosueiras/vim-terraform-completion https://github.com/juliosueiras/vim-terraform-completion, hoping to migrate to langserver by this year so any editor can use it P.S. sorry for the self-promo

gergnz 9y ago

I find that both terraform and cloudformation have their pros and cons like many things in life. It comes down to personal preference and how you think (vim/Emacs?). For me, I choose cfndsl to build my cloudformation templates. This gives me the ability to actually write code. JSON, YAML, HCL aren't full languages. Yes tf and cf try to provide language like concepts (iteration, etc) but really at the end of the day they are just a definition of your environment and not strictly code. Note: I maintain cfndsl, so I'm totally bias.

paulddraper 9y ago

> servers will be down for the next few hours. We are recreating our entire environment to replace AWS CloudFormation with @HashiCorp Terraform. One of the greatest strengths of terraform (vs say, cloudformation) is that you can adopt existing resources, and ZERO downtime is needed to migrate. Why take downtime?

stimur 9y ago
Because we needed to re-deploy our database, which unfortunately requires downtime.