5 ms·

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can

Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.

iaw 12y ago
IMO, The problem is that the manager of the cloud product isn't the person that is ordering the reboot. The manager of the cloud product considers communication with customers to mitigate their issues a primary concern. Person ordering the reboot is more 'technically' focused and representative of why engineers have a hard time as managers (eg ignoring the potential ill will this type of action can create).
toomuchtodo 12y ago
> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.
- rurounijones 12y ago
  Good point, I wonder if they didn't because that would have crippled capacity due to people rebooting, using up the patched hosts and leaving others without the ability to spin up VMs as it sounds like a significant % of hosts are bad at this point..
- rands3311 12y ago
  Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?
  - toomuchtodo 12y ago
    > Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours? Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."
    - rands3311 12y ago
      There very well could be. If this issue is related to only handful of instance types then there goes all that extra capacity.
    - nacs 12y ago
      > Is there that little slack in Amazon's compute capacity? I'm sure there's plenty of "slack" under most circumstances. However this affects the majority of users so all of them setting up new instances at the same time would likely be impossible. For the scale of this reboot, they'd have to maintain 30-50% extra capacity which would likely be financially impossible at those rates.
    - seanp2k2 12y ago
      Any slack capacity is waste, and Amazon sure hates that. Consider where the supply of the spot market comes from.
    - rodgerd 12y ago
      > Is there that little slack in Amazon's compute capacity? Amazon's ability to deliver sharp prices does not come from leaving unused tin lying around in datacentres.
      - toomuchtodo 12y ago
        Amazon markets their cloud infrastructure as having the ability to scale up at a moment's notice. If they don't have excess capacity, where am I going to scale to? Back to a colo environment?
- res0nat0r 12y ago
  This is normally what they do when they roll out forced reboots like this. Note the forum posting says there isn't guarantee of being on an updated host...that is because the patching isn't complete across the region yet.
- astral303 12y ago
  There is not enough capacity. I think Amazon did not make this decision lightly.
  - Xorlev 12y ago
    You must construct additional pylons.
- pas256 12y ago
  Stop relying on long running instances. Design your infrastructure for failure. This is not Amazon's responsibility.
  - toomuchtodo 12y ago
    Stop assuming that everyone operates at Netflix scale. Not everyone wants to watch their database, memcached, and redis instances thrash all day as instances dance around from physical box to physical box.
x0x0 12y ago
And having dealt with ec2 support, I would have expected amazon to half-ass this just the way they did. Another reason to consider owned & operated hardware. Or at least something like monthly rentals.
- kulids77 12y ago
  How would owned & operated hardware or monthly rentals be better?
  - skuhn 12y ago
    You might not be vulnerable at all (if it is a Xen problem), or you might not have to patch so urgently. More to the point, by operating your own systems, the maintenance window is set by your staff and not Amazon.
    - michaelmior 12y ago
      I'm not aware of Xen being particularly susceptible to security vulnerabilities. Any software you have on your box could end up being compromised. I'm not sure why it's less urgent if you have your own hardware. And of course the other downside is that while you set maintenance windows, you also have to do all the maintenance. Not that owning your own hardware isn't something worth consider, just not sure that's the most compelling argument.
      - skuhn 12y ago
        There are many pros and cons to EC2 vs managed hosting vs operating it yourself. I meant that in this particular case, you are likely to have fewer headaches from this issue in your own datacenter because: 1. You probably don't run Xen in the first place. 2. If you do, this vulnerability may not be critical to you. I'm guessing that it is exploitable via the other instances on a host. If you don't share hardware with other companies, then a co-tenancy exploit may not be a huge deal. 3. And if the vulnerability is critical to you, you may be able to better schedule the maintenance for your own particular business needs. Fix half your hosts, fail over, fix the other half; schedule it during your particular low traffic period. The scope of the issue is also smaller, you just have to solve the problem for your use cases rather than the massive undertaking this is for Amazon. I don't think this one incident means EC2 is bad and real hardware is good, forever and ever. There are tradeoffs to everything.

iaw 12y ago

IMO, The problem is that the manager of the cloud product isn't the person that is ordering the reboot. The manager of the cloud product considers communication with customers to mitigate their issues a primary concern. Person ordering the reboot is more 'technically' focused and representative of why engineers have a hard time as managers (eg ignoring the potential ill will this type of action can create).

toomuchtodo 12y ago

> I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.

rurounijones 12y ago
Good point, I wonder if they didn't because that would have crippled capacity due to people rebooting, using up the patched hosts and leaving others without the ability to spin up VMs as it sounds like a significant % of hosts are bad at this point..
rands3311 12y ago
Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?
- toomuchtodo 12y ago
  > Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours? Is there that little slack in Amazon's compute capacity? I would hope not! If there isn't capacity to start my instance back up, I would hope that hitting Stop would generate a dialog to the effect of "Hey there, you won't be able to start this instance back up if you stop it right now."
  - rands3311 12y ago
    There very well could be. If this issue is related to only handful of instance types then there goes all that extra capacity.
  - nacs 12y ago
    > Is there that little slack in Amazon's compute capacity? I'm sure there's plenty of "slack" under most circumstances. However this affects the majority of users so all of them setting up new instances at the same time would likely be impossible. For the scale of this reboot, they'd have to maintain 30-50% extra capacity which would likely be financially impossible at those rates.
  - seanp2k2 12y ago
    Any slack capacity is waste, and Amazon sure hates that. Consider where the supply of the spot market comes from.
  - rodgerd 12y ago
    > Is there that little slack in Amazon's compute capacity? Amazon's ability to deliver sharp prices does not come from leaving unused tin lying around in datacentres.
    - toomuchtodo 12y ago
      Amazon markets their cloud infrastructure as having the ability to scale up at a moment's notice. If they don't have excess capacity, where am I going to scale to? Back to a colo environment?
res0nat0r 12y ago
This is normally what they do when they roll out forced reboots like this. Note the forum posting says there isn't guarantee of being on an updated host...that is because the patching isn't complete across the region yet.
astral303 12y ago
There is not enough capacity. I think Amazon did not make this decision lightly.
- Xorlev 12y ago
  You must construct additional pylons.
pas256 12y ago
Stop relying on long running instances. Design your infrastructure for failure. This is not Amazon's responsibility.
- toomuchtodo 12y ago
  Stop assuming that everyone operates at Netflix scale. Not everyone wants to watch their database, memcached, and redis instances thrash all day as instances dance around from physical box to physical box.

x0x0 12y ago

And having dealt with ec2 support, I would have expected amazon to half-ass this just the way they did. Another reason to consider owned & operated hardware. Or at least something like monthly rentals.

kulids77 12y ago
How would owned & operated hardware or monthly rentals be better?
- skuhn 12y ago
  You might not be vulnerable at all (if it is a Xen problem), or you might not have to patch so urgently. More to the point, by operating your own systems, the maintenance window is set by your staff and not Amazon.
  - michaelmior 12y ago
    I'm not aware of Xen being particularly susceptible to security vulnerabilities. Any software you have on your box could end up being compromised. I'm not sure why it's less urgent if you have your own hardware. And of course the other downside is that while you set maintenance windows, you also have to do all the maintenance. Not that owning your own hardware isn't something worth consider, just not sure that's the most compelling argument.
    - skuhn 12y ago
      There are many pros and cons to EC2 vs managed hosting vs operating it yourself. I meant that in this particular case, you are likely to have fewer headaches from this issue in your own datacenter because: 1. You probably don't run Xen in the first place. 2. If you do, this vulnerability may not be critical to you. I'm guessing that it is exploitable via the other instances on a host. If you don't share hardware with other companies, then a co-tenancy exploit may not be a huge deal. 3. And if the vulnerability is critical to you, you may be able to better schedule the maintenance for your own particular business needs. Fix half your hosts, fail over, fix the other half; schedule it during your particular low traffic period. The scope of the issue is also smaller, you just have to solve the problem for your use cases rather than the massive undertaking this is for Amazon. I don't think this one incident means EC2 is bad and real hardware is good, forever and ever. There are tradeoffs to everything.