4 ms·

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 0400 PDT and check your email and stay near your laptop and internet at least once an hour, every hour, until (potentially) 0400 PDT on Monday morning. Hooray, my Sunday plans are fucked! They suggest taking backups/snapshots of your instances before the reboot window. Given the throughput required to push multi-hundred GB images from public cloud servers to Cloud Files for storage, I am willing to bet that the backup network is maxed out and will stay maxed out until the outage. I wonder if Rackspace found out about the rumored Xen exploit from the same people that told Amazon, or if Amazon told Rackspace but waited a little bit to make it more painful for Rackspace's customers...

click170 12y ago
The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released. The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?
- kam 12y ago
  The Xen pre-disclosure list is public, and Rackspace is on it: http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html They also spell out the policy for who is eligible.
  - chrissnell 12y ago
    Thanks for the link. Given that Amazon was notifying customers as early as two days ago, we can assume that Rackspace received notification from Xen at least two days ago. They waited until Friday night to announce this? Awful.
  - Maxious 12y ago
    prgmr.com also got inducted 2 days ago http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen-hypervisor.html http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen... Xen has previously had problems with people leaking embargoed security issues http://www.gossamer-threads.com/lists/xen/devel/248239 http://www.gossamer-threads.com/lists/xen/devel/248239
  - pcarmichael 12y ago
    Any word on if/when Linode may schedule reboots?
    - kbar13 12y ago
      https://forum.linode.com/viewtopic.php?p=64914#p64914 https://forum.linode.com/viewtopic.php?p=64914#p64914
csmitheu 12y ago
Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan accordingly. No sleep is lost even when something explodes at 2AM (other than for the ops team, who in this case you contracted out).
- mikegioia 12y ago
  You're missing his point, and it's condescending/naive of you to assume that his app hasn't been tested to withstand it with no information to tell you that. My app has been thoroughly tested to withstand reboots; however, I have 22 machines with rackspace and (a) I've never tested all 22 going down at random times, and (b) I STILL want to be awake in case anything happens unexpectedly. The pain here is the timing and announcement window for those of us who want to monitor their machines for problems.
  - csmitheu 12y ago
    Well when you contract out your infrastructure that really should be part of your DR plan. Analysing failure conditions is really priority one when you put your stuff on someone else's turf as you have little control over this, despite contracts etc. You did do a DR plan right and did check your contract with RackSpace? I want to be woken up if something doesn't come back, not if it does or even if it has gone into limp mode. Seriously you're looking at the wrong end of the problem.
    - mynameisvlad 12y ago
      Jesus christ dude, we get it, you're a perfect sysadmin who has clearly covered all possible bases. But others are not like you. Systems are not always 100% foolproof, people don't have comprehensive DR plans. Is it that hard for you to understand why someone may want to be up when there is a reboot? Like seriously?

click170 12y ago

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released. The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

kam 12y ago
The Xen pre-disclosure list is public, and Rackspace is on it: http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html They also spell out the policy for who is eligible.
- chrissnell 12y ago
  Thanks for the link. Given that Amazon was notifying customers as early as two days ago, we can assume that Rackspace received notification from Xen at least two days ago. They waited until Friday night to announce this? Awful.
- Maxious 12y ago
  prgmr.com also got inducted 2 days ago http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen-hypervisor.html http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen... Xen has previously had problems with people leaking embargoed security issues http://www.gossamer-threads.com/lists/xen/devel/248239 http://www.gossamer-threads.com/lists/xen/devel/248239
- pcarmichael 12y ago
  Any word on if/when Linode may schedule reboots?
  - kbar13 12y ago
    https://forum.linode.com/viewtopic.php?p=64914#p64914 https://forum.linode.com/viewtopic.php?p=64914#p64914

csmitheu 12y ago

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently. Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan accordingly. No sleep is lost even when something explodes at 2AM (other than for the ops team, who in this case you contracted out).

mikegioia 12y ago
You're missing his point, and it's condescending/naive of you to assume that his app hasn't been tested to withstand it with no information to tell you that. My app has been thoroughly tested to withstand reboots; however, I have 22 machines with rackspace and (a) I've never tested all 22 going down at random times, and (b) I STILL want to be awake in case anything happens unexpectedly. The pain here is the timing and announcement window for those of us who want to monitor their machines for problems.
- csmitheu 12y ago
  Well when you contract out your infrastructure that really should be part of your DR plan. Analysing failure conditions is really priority one when you put your stuff on someone else's turf as you have little control over this, despite contracts etc. You did do a DR plan right and did check your contract with RackSpace? I want to be woken up if something doesn't come back, not if it does or even if it has gone into limp mode. Seriously you're looking at the wrong end of the problem.
  - mynameisvlad 12y ago
    Jesus christ dude, we get it, you're a perfect sysadmin who has clearly covered all possible bases. But others are not like you. Systems are not always 100% foolproof, people don't have comprehensive DR plans. Is it that hard for you to understand why someone may want to be up when there is a reboot? Like seriously?