4 ms·

They generally send you an advance email. I just had to migrate our Jenkins server a week or two ago because of this. I received something like 15 days notice o

They generally send you an advance email. I just had to migrate our Jenkins server a week or two ago because of this. I received something like 15 days notice on that one. But obviously if there's a hard failure, they aren't always going to be able to give you the amount of time you'd want. Generally speaking, you should have accounted for this situation ahead of time in your engineering plans. Amazon EC2 doesn't have anything like vmotion, it's just a bunch of KVM virts. If you're using the GUI, the first time you try a shutdown, it will do a normal request, but then if you go back and try it again while the first request is still pending, you should see the option for doing a hard restart. Try that and give it some time. Sometimes it takes an hour or two to get through. Otherwise, Amazon's tech support can help you.

rschmitty 13y ago
> Generally speaking, you should have accounted for this situation ahead of time in your engineering plans. I believe this comes as a shock to most people the first time they receive this email, it was to us at least. When we signed up with amazon there was no guideline or advice saying "hey in a year or 2 your hardware might fail or need replaced, have a migration plan ready" Perhaps it was our naivety, but we just thought hey, its the cloud, what could go wrong?! Now of course are are battle hardened
- michaelmior 13y ago
  More accurately: "Your hardware my fail at any time."
  - rch 13y ago
    I had an instance fail (unresponsive then with that same stop/start delay) on the same day in two consecutive years. I remember because it happened to be Valentine's Day, and I had to break out the laptop for while to check on things. I always wondered if that was a completely random occurrence, or part of some maintenance schedule that randomly affected me twice. The last place I worked has a policy of making zero changes in production Friday-Sunday or around holidays. It was one of their better practices.
- m_ram 13y ago
  FWIW, they have a brief section about hardware failure in the Getting Started Guide [1]. I don't know how long it's been there. [1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-console.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-... (bottom of the page)
- cmelbye 13y ago
  I honestly thought most people knew that about EC2 as one of the core "trade offs" or engineering decision that allows the platform to be what it is, compared to a more traditional VPS provider.
  - saurik 13y ago
    That isn't a tradeoff of EC2: hardware fails... I've had servers from actual unmanaged server providers have random failures and need maintenance or need to be replaced as well. Hell: I've had my hosting provider tell me they are moving data centers before ;P. The real problem here is just that Amazon has made the idea of putting your servers in the cloud so easy that people who don't understand that the servers don't run on evil magic are able to use them.
- vidarh 13y ago
  EC2 used to terminate instances with no warning in many situations when it launched. It seems they've concluded most people didn't understand that, and avoid that whenever possible now. But "cloud" compute services should in general be treated as less reliable per individual unit unless your provider explicitly explain to you why not (such as guaranteeing to use a high-availability distributed filesystem), as you no direct way of ascertaining status of the underlying hardware. You need to plan for failure regardless.
misframer 13y ago
EC2 uses Xen.
pbiggar 13y ago
One of the reasons running jenkins on EC2 sucks for developers :( The data is stored on the machine, and there's a big risk of losing all your CI/CD infrastructure. Have you given any thought to moving to something like https://circleci.com https://circleci.com? [disclosure: I work there]
- maaku 13y ago
  That's why you use EBS and snapshots...
  - pbiggar 13y ago
    EBS is really slow for CI loads, in my experience. We deliberately don't touch it.
- mdellabitta 13y ago
  It wasn't that bad. I stopped and started an instance. Done.

rschmitty 13y ago

> Generally speaking, you should have accounted for this situation ahead of time in your engineering plans. I believe this comes as a shock to most people the first time they receive this email, it was to us at least. When we signed up with amazon there was no guideline or advice saying "hey in a year or 2 your hardware might fail or need replaced, have a migration plan ready" Perhaps it was our naivety, but we just thought hey, its the cloud, what could go wrong?! Now of course are are battle hardened

michaelmior 13y ago
More accurately: "Your hardware my fail at any time."
- rch 13y ago
  I had an instance fail (unresponsive then with that same stop/start delay) on the same day in two consecutive years. I remember because it happened to be Valentine's Day, and I had to break out the laptop for while to check on things. I always wondered if that was a completely random occurrence, or part of some maintenance schedule that randomly affected me twice. The last place I worked has a policy of making zero changes in production Friday-Sunday or around holidays. It was one of their better practices.
m_ram 13y ago
FWIW, they have a brief section about hardware failure in the Getting Started Guide [1]. I don't know how long it's been there. [1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-console.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-... (bottom of the page)
cmelbye 13y ago
I honestly thought most people knew that about EC2 as one of the core "trade offs" or engineering decision that allows the platform to be what it is, compared to a more traditional VPS provider.
- saurik 13y ago
  That isn't a tradeoff of EC2: hardware fails... I've had servers from actual unmanaged server providers have random failures and need maintenance or need to be replaced as well. Hell: I've had my hosting provider tell me they are moving data centers before ;P. The real problem here is just that Amazon has made the idea of putting your servers in the cloud so easy that people who don't understand that the servers don't run on evil magic are able to use them.
vidarh 13y ago
EC2 used to terminate instances with no warning in many situations when it launched. It seems they've concluded most people didn't understand that, and avoid that whenever possible now. But "cloud" compute services should in general be treated as less reliable per individual unit unless your provider explicitly explain to you why not (such as guaranteeing to use a high-availability distributed filesystem), as you no direct way of ascertaining status of the underlying hardware. You need to plan for failure regardless.

misframer 13y ago

EC2 uses Xen.

pbiggar 13y ago

One of the reasons running jenkins on EC2 sucks for developers :( The data is stored on the machine, and there's a big risk of losing all your CI/CD infrastructure. Have you given any thought to moving to something like https://circleci.com https://circleci.com? [disclosure: I work there]

maaku 13y ago
That's why you use EBS and snapshots...
- pbiggar 13y ago
  EBS is really slow for CI loads, in my experience. We deliberately don't touch it.
mdellabitta 13y ago
It wasn't that bad. I stopped and started an instance. Done.