6 ms·

My advice is: don't keep your eggs in one basket. AZs a localised redundancy, but as Cloud is cheap and plentiful, you should be using two or more regions, at l

by movedx 10y ago

My advice is: don't keep your eggs in one basket. AZs a localised redundancy, but as Cloud is cheap and plentiful, you should be using two or more regions, at least, to house your solution (if it's important to you.) EDIT: less arrogant. I need a coffee.

jacquesm 10y ago
Two different vendors if you can afford it. It's a bit of a hassle though.
- movedx 10y ago
  I like to stick to one, but I have seen some success stories with an AWS/GCE mix :-) HashiCorp's Terraform makes it a lot easier to go multi Cloud, and abstracting away configuration of the OS and applications/state with Ansible makes the whole process a lot easier too.
  - cfieber 10y ago
    Here is one success story: https://cloudplatform.googleblog.com/2017/02/guest-post-multi-cloud-continuous-delivery-using-Spinnaker-at-Waze.html https://cloudplatform.googleblog.com/2017/02/guest-post-mult...
gamache 10y ago
But now you're talking about added effort. Multi-AZ on AWS is easy and fairly automatic, multi-region (and multi-provider) not so much. It's easy to say things like this, but people who can do ops are not cheap and plentiful.
- movedx 10y ago
  The only difficult aspect of multi-region use is data replication, which I can confirm is a (somewhat) difficult problem. This issue was with S3 which has an option to automatically replicate data from the bucket's region to another one. It's a check box. A simple bit of logic in the application and you can move between regions with ease. Even data replication has options for this, too. And I work in Ops.
  - gamache 10y ago
    Well, you've explained how to do multi-region in S3. Now let's cover EC2, ELB, EBS, VPC, RDS, Lambda, ElastiCache, API Gateway, and all the other bits of AWS that make up my services. And then we can move on to failover application logic.
    - movedx 10y ago
      I picked out S3 as this issue is directly related to it, yet the solution is simple: turn on replication and have your application work with it (which is on the developers, not ops.) EC2: why are you replicating EC2 instances or AMIs across regions? Why aren't you using build tools to automatically create AMIs for you out of your CI processes? ELB: Eh? Why do I need ELBs to be multi-regional? I'm a little confused by this on, sorry. EBS: My systems tend to be stateless, storing as much log, audit, or data in external systems such as RDS, DynamoDB, S3, etc. Storing things on the local system's storage is a bit risky, but if you have to there are disk replication solutions available. EFS comes to mind for making that easier. Backups also come to mind in the event of data loss. VPC: Why does a VPC need to be cross regional? This one is also lost on me. RDS: Replication is easy -- it's done for you. Convincing developers their application needs to potentially work with a backup endpoint to the data is harder than data replication problems at times. More often than not, it's simply a case of switching to a read-only mode whilst you recover the write copy of your RDS instance, but this is the role of the developers, not ops. Lambda, ElastiCache, API Gateway... all these things aren't arguments against my original point: architect correctly. Yes it involves more work (from the developer's perspective, mostly), but more often than not in the event of a failure you're left head and shoulders above your nearest competition and left soaking up the profits as a result. Based on your responses, however, I think we can safely agree to disagree and move on. Have a great day! I hope you weren't too badly effected by the S3 outage! EDIT: typo.
      - webo 10y ago
        >EC2: why are you replicating EC2 instances or AMIs across regions? Exactly to avoid single region outages?

by movedx 10y ago

jacquesm 10y ago

Two different vendors if you can afford it. It's a bit of a hassle though.

movedx 10y ago
I like to stick to one, but I have seen some success stories with an AWS/GCE mix :-) HashiCorp's Terraform makes it a lot easier to go multi Cloud, and abstracting away configuration of the OS and applications/state with Ansible makes the whole process a lot easier too.
- cfieber 10y ago
  Here is one success story: https://cloudplatform.googleblog.com/2017/02/guest-post-multi-cloud-continuous-delivery-using-Spinnaker-at-Waze.html https://cloudplatform.googleblog.com/2017/02/guest-post-mult...

gamache 10y ago

But now you're talking about added effort. Multi-AZ on AWS is easy and fairly automatic, multi-region (and multi-provider) not so much. It's easy to say things like this, but people who can do ops are not cheap and plentiful.

movedx 10y ago
The only difficult aspect of multi-region use is data replication, which I can confirm is a (somewhat) difficult problem. This issue was with S3 which has an option to automatically replicate data from the bucket's region to another one. It's a check box. A simple bit of logic in the application and you can move between regions with ease. Even data replication has options for this, too. And I work in Ops.
- gamache 10y ago
  Well, you've explained how to do multi-region in S3. Now let's cover EC2, ELB, EBS, VPC, RDS, Lambda, ElastiCache, API Gateway, and all the other bits of AWS that make up my services. And then we can move on to failover application logic.
  - movedx 10y ago
    I picked out S3 as this issue is directly related to it, yet the solution is simple: turn on replication and have your application work with it (which is on the developers, not ops.) EC2: why are you replicating EC2 instances or AMIs across regions? Why aren't you using build tools to automatically create AMIs for you out of your CI processes? ELB: Eh? Why do I need ELBs to be multi-regional? I'm a little confused by this on, sorry. EBS: My systems tend to be stateless, storing as much log, audit, or data in external systems such as RDS, DynamoDB, S3, etc. Storing things on the local system's storage is a bit risky, but if you have to there are disk replication solutions available. EFS comes to mind for making that easier. Backups also come to mind in the event of data loss. VPC: Why does a VPC need to be cross regional? This one is also lost on me. RDS: Replication is easy -- it's done for you. Convincing developers their application needs to potentially work with a backup endpoint to the data is harder than data replication problems at times. More often than not, it's simply a case of switching to a read-only mode whilst you recover the write copy of your RDS instance, but this is the role of the developers, not ops. Lambda, ElastiCache, API Gateway... all these things aren't arguments against my original point: architect correctly. Yes it involves more work (from the developer's perspective, mostly), but more often than not in the event of a failure you're left head and shoulders above your nearest competition and left soaking up the profits as a result. Based on your responses, however, I think we can safely agree to disagree and move on. Have a great day! I hope you weren't too badly effected by the S3 outage! EDIT: typo.
    - webo 10y ago
      >EC2: why are you replicating EC2 instances or AMIs across regions? Exactly to avoid single region outages?