8 ms·

Auto Recovery for Amazon EC2

167 points by tshtf 12y ago

oellegaard 12y ago
Heavy EC2 user here. This doesn't solve your problems, if you want to do this right, setup an EC2 Auto Scaling group and build an image each time you need to change your server. That is the proven way most large deployments work, including Netflix.
- oellegaard 12y ago
  I have a semi-old blog post about how we do it here: http://blog.kristian.io/aws/2014/06/28/how-we-moved-from-heroku-to-aws-and-improved-our-setup.html http://blog.kristian.io/aws/2014/06/28/how-we-moved-from-her...
- jontro 12y ago
  Is there any guide on how to integrate this in a deployment workflow? I would be very much interested in reading up on how to do this the best way.
  - lebski88 12y ago
    We use this approach at MixRadio, you can read about it on our blog: http://dev.mixrad.io/blog/2014/10/31/How-we-deploy-at-MixRadio/ http://dev.mixrad.io/blog/2014/10/31/How-we-deploy-at-MixRad... There is also a video of a talk about it: https://skillsmatter.com/skillscasts/6057-herding-cattle-with-clojure-at-mixradio https://skillsmatter.com/skillscasts/6057-herding-cattle-wit... We based our approach on Netflix but ended up building our own tools which we've now open sourced.
  - oellegaard 12y ago
    If you're interested, shoot me an email and I will be happy to go through our setup. You can find my contact info at kristian.io
- discodave 12y ago
  Yes but most people don't have large deployments like Netflix. The main reason Neftlix and co build entire images is because if you're scaling up and down by thousands of instances then the extra time for those instances to do all the build, configuration and so on is material. So you're better to do it once and essentially CTRL+C CTRL+V. If you're only scaling up a few instances at a time then your number one consideration is probably making life easy for yourself.
  - bkirkbri 12y ago
    We tend to take an in-between approach, baking base AMIs as needed but configuring and installing code with Ansible at boot time. We've got scripts to cut tarballs with an Ansible playbook plus support files from any tag in our repo. Then it's just a matter of specifying the tag in user-data when starting an instance. We do that through CloudFormation. edit: grammar
bkeroack 12y ago
At the risk of being down voted, let me say that this is yet another AWS "feature" that is primarily a workaround for deficiencies in the platform.
- waitwaitwhay 12y ago
  Can you explain that for me? I use Microsoft Azure a lot and there things like this just happen automatically. No configurations, alarms, etc.
- regularfry 12y ago
  If by "the platform" you mean "available computing hardware", then yes. I'm not sure that's a useful data point.
  - bkeroack 12y ago
    In the narrowest sense your sentence is correct. Perhaps you mean that you think this can be expected on any computing hardware, which is far from correct. If all you've ever used has been public cloud services you can be forgiven for having this misconception.
    - regularfry 12y ago
      "Available" meaning "reasonable to expect to support a userbase the size of EC2's with." If you start with "gotta run standardish x86_64 Red Hat or Ubuntu by the million or so" and work outwards from there, you're not really in a space where bulletproof hardware looks tempting. VCPU lock-stepping might, though.
tedunangst 12y ago
Wouldn't transparent migration to new hardware be even better? Isn't one of the advantages of virtualization the ability to move a running image from one machine to another?
- chacham15 12y ago
  In many cases that is not possible. If the machine loses network connectivity, has a disk error, is infinite looping, is out of memory, etc. how is another machine supposed to access its data?
deleted 12y ago
[deleted]
fletchowns 12y ago
An important note if you want to use this right away: This feature is currently available for the C3, C4, M3, R3, and T2 instance types running in the US East (Northern Virginia) region; we plan to make it available in other regions as quickly as possible. The instances must be running within a VPC, must use EBS-backed storage, but cannot be Dedicated Instances.
- waskosky 12y ago
  For people not yet on one of the newer instance types (HVM VPC), it is possible to migrate directly assuming you can tolerate a few minutes of downtime to make the final switch. This was the guide that helped me: https://forums.aws.amazon.com/thread.jspa?threadID=155526 https://forums.aws.amazon.com/thread.jspa?threadID=155526
- wahnfrieden 12y ago
  VPC-only sounds like a giant caveat, and it is, but this is a good opportunity to note that this is the trend now with AWS and the direction they're heading - (non-VPC) "EC2 Classic" is being gradually phased out, VPC is now the default for new accounts, and most new features are being added only to VPC. So, time for everyone to start thinking about migrating.
  - pjl 12y ago
    AWS recently announced ClassicLink [1], which helps with migration: "In order to allow EC2-Classic instances to communicate with these resources, we are introducing a new feature known as ClassicLink. You can now enable this feature for any or all of your VPCs and then put your existing Classic instances in to VPC security groups." [1] https://aws.amazon.com/blogs/aws/classiclink-private-communication-between-classic-ec2-instances-vpc-resources/ https://aws.amazon.com/blogs/aws/classiclink-private-communi...
  - mbell 12y ago
    > this is the trend now with AWS and the direction they're heading - (non-VPC) "EC2 Classic" is being gradually phased out New AWS accounts can't even use EC2 Classic, it's effectively deprecated at this point.
saryant 12y ago
I've been having a lot of issues with r3.large instances becoming unreachable lately. Hoping this can serve as a stopgap.
- jeffbarr 12y ago
  Have you noted your problem in the EC2 Forum or consulted AWS Support?
  - saryant 12y ago
    edit: I was wrong. Ignore what was here before, AWS did respond and I just didn't notice. Apologies to AWS for speaking out of turn.
    - jeffbarr 12y ago
      Huh, that's no good. Can you email me (address is in profile) and I'll see what's going on?
      - saryant 12y ago
        I spoke without double-checking our account. We did receive a response about the forum creation problem and that was the result of my own misunderstanding. I'll update my original post.
bmurphy1976 12y ago
Please note that this is for EBS backed instances only. If you want something similar for ephemeral instances, do what we do: min 1 max 1 auto scaling groups. We've found that Amazon is pretty good at catching bad instances and terminating them, although on occasion we do have to terminate an instance manually. The autoscaling group takes care of the rest.
- deleted 12y ago
  [deleted]
biot 12y ago
Any reason why this isn't automatic? From the "Recover your instance" docs: Examples of problems that cause system status checks to fail include: * Loss of network connectivity * Loss of system power * Software issues on the physical host * Hardware issues on the physical host All of these are on the physical host, which end users cannot control. So if AWS has an issue that kills your VM, if you don't have this setup then your instance is effectively dead?
- moe 12y ago
  Any reason why this isn't automatic? Perhaps it's a co-promotion for CloudWatch. I would guess quite a few of their users had never heard of or seen a use for CloudWatch. Some of them might now enable "detailed monitoring" for $3.50/mo per instance while they're at it.
- perlgeek 12y ago
  Loss of network connectivity sounds like it could be temporary. If you have long-running calculations and want to wait for the result, it might make sense to wait a bit longer and see if the network comes up eventually. And there's no indication that the hardware and software "issues" are permanent or even fatal.
  - biot 12y ago
    The strategy of "wait a bit longer" can be configured with an exact value for how long to wait via this recovery feature. However, given that you don't have any information about the failure's permanence or whether it can be resolved by AWS staff without taking the host down I don't see why waiting indefinitely is a particularly good option. In some ways, I guess this answers my own question. Amazon doesn't know how long you might want to wait or if you have a VM that you would even want to have recovered, so configuring this lets you tell Amazon what your parameters around recovery should be.
- larrymcp 12y ago
  You know, it's funny but reading the EC2 forums you see several occasions where Amazon does reboot instances automatically after a failure. I recall several posts where a customer asked a question along the lines of "Why did my instance reboot" and then someone from Amazon replied something to the effect of "Sorry there was an issue with the underlying hardware but we did restart your server on new hardware".
CarlHoerberg 12y ago
Finally, it's crazy that they've haven't implemented this earlier, and why isn't it enabled by default, like on GCE? We've had for a long time an app that just polls the ec2 api and looks for impaired instances and then automatically restarts them. We have about 2-10 impaired/scheduled-for-reboot/on-deprecated hardware-instance per month so that app is quite a time-saver.
alrs 12y ago
The ugly caveat isn't VPC, it's EBS. This lands on the wrong side of pets-versus-cattle. AWS has been moving towards giving people what they want, but it's still best practice to use ephemeral storage and architect accordingly.
- toomuchtodo 12y ago
  > but it's still best practice to use ephemeral storage and architect accordingly Its not worth the engineer time. Use EBS volumes, clean them up when they're no longer in use after termination. The only time you need local/ephemeral storage is swap or scratch space, or throughput you can't get from general or provisioned EBS. Plus, you get auto recovery now without having to have architected for it ;)
  - gtaylor 12y ago
    After I say this, they'll probably have a huge EBS-caused outage, but it feels like EBS is much more capable now than it was even a year or two ago. The General Purpose SSDs and the Provisioned IOPS have more than handled our performance concerns. Since that last awful multi-AZ outage a year or two ago, there hasn't been much to deal with for us.
  - gst 12y ago
    EBS is great if it works for requirements, but unfortunately the maximum size of an EBS volume is limited to only 1 TB. With SSD instance storage you can get up to 6.4 TB.
    - Rapzid 12y ago
      Fortunately AWS is releasing 16TB volumes in the near future :)
- Rapzid 12y ago
  Here is a fun story. Amazon's RDS services are backed by EBS. They also restart the instances automatically if they encounter issues. But sure, "Best PracticesTM".
kolev 12y ago
If you rely on something like this, you rely on nothing. This is like crutches for your broken architecture. For singleton roles, you could do an autoscaling group of one and do better.
- kolev 12y ago
  Not sure why the downvotes - at least ASGs can be expanded later unlike the one-offs. Plus, you should never have SPOFs anyway. There was a comment below about not all projects being of Netflix' scale - well, there's Digital Ocean for the smaller one - EC2 is not the most cost-effective solution for small projects anyway.
halayli 12y ago
This makes me so happy.
j-kidd 12y ago
This shall be a great fit for the NAT/Bastion instance, since the high-availability setup has a few drawbacks: https://aws.amazon.com/articles/2781451301784570 https://aws.amazon.com/articles/2781451301784570
andr 12y ago
I think CodeDeploy is quite an undervalued AWS tool. It's a combination of Puppet for server config and Heroku-style deploys. Together with AutoScaling it makes it trivial to set up any number of identical servers, without relying on custom AMIs or recovery.

167 points by tshtf 12y ago

Heavy EC2 user here. This doesn't solve your problems, if you want to do this right, setup an EC2 Auto Scaling group and build an image each time you need to change your server. That is the proven way most large deployments work, including Netflix.

oellegaard 12y ago
I have a semi-old blog post about how we do it here: http://blog.kristian.io/aws/2014/06/28/how-we-moved-from-heroku-to-aws-and-improved-our-setup.html http://blog.kristian.io/aws/2014/06/28/how-we-moved-from-her...
jontro 12y ago
Is there any guide on how to integrate this in a deployment workflow? I would be very much interested in reading up on how to do this the best way.
- lebski88 12y ago
  We use this approach at MixRadio, you can read about it on our blog: http://dev.mixrad.io/blog/2014/10/31/How-we-deploy-at-MixRadio/ http://dev.mixrad.io/blog/2014/10/31/How-we-deploy-at-MixRad... There is also a video of a talk about it: https://skillsmatter.com/skillscasts/6057-herding-cattle-with-clojure-at-mixradio https://skillsmatter.com/skillscasts/6057-herding-cattle-wit... We based our approach on Netflix but ended up building our own tools which we've now open sourced.
- oellegaard 12y ago
  If you're interested, shoot me an email and I will be happy to go through our setup. You can find my contact info at kristian.io
discodave 12y ago
Yes but most people don't have large deployments like Netflix. The main reason Neftlix and co build entire images is because if you're scaling up and down by thousands of instances then the extra time for those instances to do all the build, configuration and so on is material. So you're better to do it once and essentially CTRL+C CTRL+V. If you're only scaling up a few instances at a time then your number one consideration is probably making life easy for yourself.
- bkirkbri 12y ago
  We tend to take an in-between approach, baking base AMIs as needed but configuring and installing code with Ansible at boot time. We've got scripts to cut tarballs with an Ansible playbook plus support files from any tag in our repo. Then it's just a matter of specifying the tag in user-data when starting an instance. We do that through CloudFormation. edit: grammar

bkeroack 12y ago

At the risk of being down voted, let me say that this is yet another AWS "feature" that is primarily a workaround for deficiencies in the platform.

waitwaitwhay 12y ago
Can you explain that for me? I use Microsoft Azure a lot and there things like this just happen automatically. No configurations, alarms, etc.
regularfry 12y ago
If by "the platform" you mean "available computing hardware", then yes. I'm not sure that's a useful data point.
- bkeroack 12y ago
  In the narrowest sense your sentence is correct. Perhaps you mean that you think this can be expected on any computing hardware, which is far from correct. If all you've ever used has been public cloud services you can be forgiven for having this misconception.
  - regularfry 12y ago
    "Available" meaning "reasonable to expect to support a userbase the size of EC2's with." If you start with "gotta run standardish x86_64 Red Hat or Ubuntu by the million or so" and work outwards from there, you're not really in a space where bulletproof hardware looks tempting. VCPU lock-stepping might, though.

tedunangst 12y ago

Wouldn't transparent migration to new hardware be even better? Isn't one of the advantages of virtualization the ability to move a running image from one machine to another?

chacham15 12y ago
In many cases that is not possible. If the machine loses network connectivity, has a disk error, is infinite looping, is out of memory, etc. how is another machine supposed to access its data?

deleted 12y ago

[deleted]

fletchowns 12y ago

An important note if you want to use this right away: This feature is currently available for the C3, C4, M3, R3, and T2 instance types running in the US East (Northern Virginia) region; we plan to make it available in other regions as quickly as possible. The instances must be running within a VPC, must use EBS-backed storage, but cannot be Dedicated Instances.

waskosky 12y ago
For people not yet on one of the newer instance types (HVM VPC), it is possible to migrate directly assuming you can tolerate a few minutes of downtime to make the final switch. This was the guide that helped me: https://forums.aws.amazon.com/thread.jspa?threadID=155526 https://forums.aws.amazon.com/thread.jspa?threadID=155526
wahnfrieden 12y ago
VPC-only sounds like a giant caveat, and it is, but this is a good opportunity to note that this is the trend now with AWS and the direction they're heading - (non-VPC) "EC2 Classic" is being gradually phased out, VPC is now the default for new accounts, and most new features are being added only to VPC. So, time for everyone to start thinking about migrating.
- pjl 12y ago
  AWS recently announced ClassicLink [1], which helps with migration: "In order to allow EC2-Classic instances to communicate with these resources, we are introducing a new feature known as ClassicLink. You can now enable this feature for any or all of your VPCs and then put your existing Classic instances in to VPC security groups." [1] https://aws.amazon.com/blogs/aws/classiclink-private-communication-between-classic-ec2-instances-vpc-resources/ https://aws.amazon.com/blogs/aws/classiclink-private-communi...
- mbell 12y ago
  > this is the trend now with AWS and the direction they're heading - (non-VPC) "EC2 Classic" is being gradually phased out New AWS accounts can't even use EC2 Classic, it's effectively deprecated at this point.

saryant 12y ago

I've been having a lot of issues with r3.large instances becoming unreachable lately. Hoping this can serve as a stopgap.

jeffbarr 12y ago
Have you noted your problem in the EC2 Forum or consulted AWS Support?
- saryant 12y ago
  edit: I was wrong. Ignore what was here before, AWS did respond and I just didn't notice. Apologies to AWS for speaking out of turn.
  - jeffbarr 12y ago
    Huh, that's no good. Can you email me (address is in profile) and I'll see what's going on?
    - saryant 12y ago
      I spoke without double-checking our account. We did receive a response about the forum creation problem and that was the result of my own misunderstanding. I'll update my original post.

bmurphy1976 12y ago

Please note that this is for EBS backed instances only. If you want something similar for ephemeral instances, do what we do: min 1 max 1 auto scaling groups. We've found that Amazon is pretty good at catching bad instances and terminating them, although on occasion we do have to terminate an instance manually. The autoscaling group takes care of the rest.

deleted 12y ago
[deleted]

biot 12y ago

Any reason why this isn't automatic? From the "Recover your instance" docs: Examples of problems that cause system status checks to fail include: * Loss of network connectivity * Loss of system power * Software issues on the physical host * Hardware issues on the physical host All of these are on the physical host, which end users cannot control. So if AWS has an issue that kills your VM, if you don't have this setup then your instance is effectively dead?

moe 12y ago
Any reason why this isn't automatic? Perhaps it's a co-promotion for CloudWatch. I would guess quite a few of their users had never heard of or seen a use for CloudWatch. Some of them might now enable "detailed monitoring" for $3.50/mo per instance while they're at it.
perlgeek 12y ago
Loss of network connectivity sounds like it could be temporary. If you have long-running calculations and want to wait for the result, it might make sense to wait a bit longer and see if the network comes up eventually. And there's no indication that the hardware and software "issues" are permanent or even fatal.
- biot 12y ago
  The strategy of "wait a bit longer" can be configured with an exact value for how long to wait via this recovery feature. However, given that you don't have any information about the failure's permanence or whether it can be resolved by AWS staff without taking the host down I don't see why waiting indefinitely is a particularly good option. In some ways, I guess this answers my own question. Amazon doesn't know how long you might want to wait or if you have a VM that you would even want to have recovered, so configuring this lets you tell Amazon what your parameters around recovery should be.
larrymcp 12y ago
You know, it's funny but reading the EC2 forums you see several occasions where Amazon does reboot instances automatically after a failure. I recall several posts where a customer asked a question along the lines of "Why did my instance reboot" and then someone from Amazon replied something to the effect of "Sorry there was an issue with the underlying hardware but we did restart your server on new hardware".

CarlHoerberg 12y ago

Finally, it's crazy that they've haven't implemented this earlier, and why isn't it enabled by default, like on GCE? We've had for a long time an app that just polls the ec2 api and looks for impaired instances and then automatically restarts them. We have about 2-10 impaired/scheduled-for-reboot/on-deprecated hardware-instance per month so that app is quite a time-saver.

alrs 12y ago

The ugly caveat isn't VPC, it's EBS. This lands on the wrong side of pets-versus-cattle. AWS has been moving towards giving people what they want, but it's still best practice to use ephemeral storage and architect accordingly.

toomuchtodo 12y ago
> but it's still best practice to use ephemeral storage and architect accordingly Its not worth the engineer time. Use EBS volumes, clean them up when they're no longer in use after termination. The only time you need local/ephemeral storage is swap or scratch space, or throughput you can't get from general or provisioned EBS. Plus, you get auto recovery now without having to have architected for it ;)
- gtaylor 12y ago
  After I say this, they'll probably have a huge EBS-caused outage, but it feels like EBS is much more capable now than it was even a year or two ago. The General Purpose SSDs and the Provisioned IOPS have more than handled our performance concerns. Since that last awful multi-AZ outage a year or two ago, there hasn't been much to deal with for us.
- gst 12y ago
  EBS is great if it works for requirements, but unfortunately the maximum size of an EBS volume is limited to only 1 TB. With SSD instance storage you can get up to 6.4 TB.
  - Rapzid 12y ago
    Fortunately AWS is releasing 16TB volumes in the near future :)
Rapzid 12y ago
Here is a fun story. Amazon's RDS services are backed by EBS. They also restart the instances automatically if they encounter issues. But sure, "Best PracticesTM".

kolev 12y ago

If you rely on something like this, you rely on nothing. This is like crutches for your broken architecture. For singleton roles, you could do an autoscaling group of one and do better.

kolev 12y ago
Not sure why the downvotes - at least ASGs can be expanded later unlike the one-offs. Plus, you should never have SPOFs anyway. There was a comment below about not all projects being of Netflix' scale - well, there's Digital Ocean for the smaller one - EC2 is not the most cost-effective solution for small projects anyway.

halayli 12y ago

This makes me so happy.

j-kidd 12y ago

This shall be a great fit for the NAT/Bastion instance, since the high-availability setup has a few drawbacks: https://aws.amazon.com/articles/2781451301784570 https://aws.amazon.com/articles/2781451301784570

andr 12y ago

I think CodeDeploy is quite an undervalued AWS tool. It's a combination of Puppet for server config and Heroku-style deploys. Together with AutoScaling it makes it trivial to set up any number of identical servers, without relying on custom AMIs or recovery.