8 ms·

Tell HN: Rackspace apparently lost some storage volumes

134 points by sz4kerto 6y ago

First, RS has informed us that they're migrating some volumes and that'll require downtime: 1st of Oct: "The above volume has been selected as available to clone and migrate to prevent being affected by the maintenance listed below. If the volume migration is not completed before October 16 at 22:00 CST, it will be affected by the below work. New volumes and clones of existing volumes will be placed in the new datacenter location and will not be affected by the below work." 5 hours ago: "Unfortunately you have one or more volumes that we have not yet been able to bring back online. The following volumes are impacted by this issue:" Then a few minutes ago: "Following extensive troubleshooting we have been unable to bring the host server on which your volume is hosted back online and as such we are unable to recover data for your device. You have the option to deploy a new Cloud Block Storage device."

wernerb 6y ago
This is incredibly frustrating, I can sympathize. But this is why we do backups even on cloud storage. I had the same thing happen with an EBS volume on AWS becoming unresponsive, unable to attach and load.
jtagen 6y ago
I'm surprised, always thought the Rackspace premium price was to prevent or significantly reduce these issues. That being said, I've had quite a few problems with them over the last year or so that is making me reconsider. The first-tier support they now outsource to India is frustrating. CloudFiles entries being "inaccessible" after saving for no known reason, happens a couple times a month. Had a managed MySQL slave replica down for 5 hours because they changed the IP of the master (mandatory maintenance) and failed to update the replica, no way to do this on my own. AWS and Azure might not be better, but they're definitely cheaper.
- folkhack 6y ago
  > I'm surprised, always thought the Rackspace premium price was to prevent or significantly reduce these issues. I've found that these systems are often there to make the business-minded customer feel better vs. offering up any sort of highly-competent technical support. Rackspace fits this bill for me although I haven't been their customer for close to 10 years due to how badly they've burnt me. On the contrary, free-tier Linode support has always been incredibly technically competent and has never let me down for the 10+ years I've been with them. If anyone from Linode is reading this - you guys are awesome, and your support staff is amazing. I can't say enough nice things about them as they've done so much to support me for a solid decade! All I'm saying is paying for support vs. not paying doesn't mean you'll have any more technically competent help in my real-world experience. It just means "you're paying for support" on paper, and typically is just an up-sell from legacy hosting providers like Rackspace or GoDaddy. Free from Linode is 100X paid from Rackspace - all depends on the company.
  - joshribakoff 6y ago
    I moved a company off Linode a few years ago because of ongoing intermittent packet loss (think 93%) for subsets of our customers, they never were able to do anything about it and just pointed fingers at the ISPs, who of course I have no direct contact or relationship with. Wouldn’t recommend.
  - snowwrestler 6y ago
    Rackspace support is not just “help you when you are having trouble,” they will do sysadmin tasks for you, basically as much as you want. It’s more like light professional services + typical support. That’s why it costs more.
    - folkhack 6y ago
      Yes, well aware. In-industry we call this "managed" and "un-managed" hosting. All of my issues actually happened with "managed" hosting 10 years ago. I was in my early-20's and still getting my feet underneath me as a Linux administrator so having a backup was something I advocated when I was provisioning our hosting. Frankly, it was probably pretty good for career development because their support staff was clueless with anything Linux administration, and if I needed an answer quick I was my only trustworthy resource. Like - basic LAMP (Linux, Apache2, MySQL, PHP) was completely beyond them even though that was the bread and butter of their business. I haven't sprung for "managed" hosting ever since because it just taught me that I was the best person to handle my interests... and it lit a fire under my butt to get way better at my Linux chops!
- berns 6y ago
  > AWS and Azure might not be better, but they're definitely cheaper. Azure has "ZERO % Annualized Failure Rate". It gives me such a cozy feeling.
  - simonswords82 6y ago
    Amen. We moved to Azure after a DDOS attack which took out our Rackspace instances and we have never looked back.
- simonswords82 6y ago
  > AWS and Azure might not be better, but they're definitely cheaper. I can't speak for AWS as we don't use it but as a former client of Rackspace and a current client of Azure I can assure you that Azure is better and cheaper by a country mile.
kbar13 6y ago
yeah hardware issues happen, it's unfortunate but it happens on a very regular basis. backups are important
- bartvk 6y ago
  On one hand, I agree with you. On the other hand, you say this is a hardware issue but do you know for a fact that is what happened? Migration sounds to me like you migrate data from one state to the other state, then delete the old state. To me, it would be logical that before you delete the old state, you verify whether the migration was successful.
  - kbar13 6y ago
    providers like this dont migrate data for no reason. there was probably a bad drive in the array or something, or some other known defect, so they are opting to migrate off the host before things get worse
- tinus_hn 6y ago
  The idea behind cloud services is that you pay someone to manage that.
  - znpy 6y ago
    Yup it's ironic however how in the end you have to manage that stuff yourself anyway.
    - tetha 6y ago
      It depends how the vendor presents and handles it. I'm not aware how RS promises look. However, for example, the hetzner cloud has an explicit choice between triple-redundant ceph-based storage for VMs, or hypervisor local SSD storage. That's good and explicit. I can make an informed choice about speed or durability.
      - 6y ago
folkhack 6y ago
This is completely anecdotal: If you find yourself on Rackspace hosting I would absolutely migrate to something more mature. Personally I've had great luck with Digital Ocean and Linode for basic/affordable VPS hosting, with AWS being a great cloud provider with an incredible tool-box at your disposal. Hell, GPC, Azure... there's just so many better options. IMO Rackspace has been on a steady decline for years. They were a choice provider back in the early 2000's when a managed/unmanaged beige box/shared cPanel/Plesk hosting was common. Back then their support was incredible. Sometime around 2010 though things started to go down hill, and I experienced more and more support issues with them. These would range from annoying all the way to "that shouldn't happen" (like losing volumes, which happened to me). The final straw was when a dedicated server was turned down by mistake, causing a major outage for 20+ of my clients at the time. This was 10 years ago and I've never looked back. They did a pretty good job at turning me into an advocate against their services. Oh and the Slicehost transition was crap too. That sucked to get over to a first-gen VPS provider only to have it sucked right back into the beast that you were trying to escape from in the first place... Sorry for the mini rant. I'm still not over the PTSD that Rackspace has caused me.
- ericcholis 6y ago
  Currently, to delete a cloud server, you're required to call support. The option to delete servers via the API and interface have been removed. On more than one occasion I've been pitched by the "support engineer" to sign up for something akin to reserved instance pricing. They claim it's a security measure. I've never had an issue with service or availability. But, the price per performance on the VM instances is so much better elsewhere. Was working to migrate to DO, but now I'm fast tracking that process.
  - folkhack 6y ago
    > Currently, to delete a cloud server, you're required to call support. This is insanity to me. I'd leave a VPS/cloud provider so fast if they started this. Even back in the day I always had the option to delete a Rackspace Cloud instance. Wow. Just wow.
  - 6y ago
seneca 6y ago
Rackspace was sold to a private equity company in late 2016. Most of their senior staff has jumped ship since then, and services have been decreasing in quality. It's essentially a different company, and not one I'd host anything important on.
- znpy 6y ago
  Ha! I've been through something similar (company being sold). According to my experience, a company that passes from an owner to another one is basically another, completely different company.
- x86_64Ubuntu 6y ago
  Is there anyone that gets bought out by a PE company that doesn't go to absolute shit? QlikTech, Sencha, TravisCI all got bought and suffered a marked decline in service and roadmap confidence. Nowadays, when I hear someone is bought by a PE firm, I start looking for the exits.
  - tedivm 6y ago
    I remember when LimeLight Networks got bought out by PE and their services turned to crap in about six months. It got so bad we got lawyers involved to break our contract with them.
  - hn_throwaway_99 6y ago
    I really wonder about this when it comes to the longer term viability of tech-focused PE firms. The following seem to be absolutely truisms: 1. The second a company is bought out by PE, all the good employees run for the exits. Corporate America isn't exactly known for its empathy and compassion, but working for a PE-owned company means you can be sure the ownership gives 0 shits about anyone working at the company. I have never heard anything but horror stories of those working for PE-owned shops. 2. Given #1, it means the products of said companies always turn to shit pretty fast. This makes me really wonder about how PE firms can really work their financial magic in the long term. Perhaps they're just better at extracting value out of companies that are already declining in the first place, so they just suck the blood dry before they discard of the carcass?
rshnotsecure 6y ago
A few months ago I found that Rackspace had left open their prod logging system for their global load balancer / firewall system (known as BlueFlood). Reporting the open Elasticsearch server to then was one of the most bizarre experiences of my life. Ultimately the system was exposed for about 2 weeks. You can find some screenshots of what their backend looked like, which came in at about 3 TB daily, here: https://blog.12security.com/rackspace-bluefood-breach/ https://blog.12security.com/rackspace-bluefood-breach/
deleted 6y ago
[deleted]
QUFB 6y ago
Is this news? This is a block storage failure on Rackspace. EBS volumes on Amazon fail all the time, as anyone who manages a large number of instances probably knows (0.1%-0.2% per year for normal volumes): > Amazon EBS offers a higher durability volume (io2), that is designed to provide 99.999% durability with an annual failure rate (AFR) of 0.001%, where failure refers to a complete or partial loss of the volume. For example, if you have 100,000 EBS io2 volumes running for 1 year, you should expect only one io2 volume to experience a failure. This makes io2 ideal for business-critical applications such as SAP HANA, Oracle, Microsoft SQL Server and IBM DB2 that will benefit from higher uptime. io2 volumes are 2000 times more reliable than typical commodity disk drives, which fail with an AFR of around 2%. All other Amazon EBS volumes are designed to provided 99.8%-99.9% durability with an AFR of between 0.1% - 0.2%, https://aws.amazon.com/ebs/features/ https://aws.amazon.com/ebs/features/
- deleted 6y ago
  [deleted]
toast0 6y ago
> First, RS has informed us that they're migrating some volumes and that'll require downtime. I'm not trying to blame you, just trying to figure out what happened. Reading October 1st message, it didn't sound like they were going to clone and migrate it, but that they were letting you know something you could do to avoid the below work (which you didn't quote). What was the description of the work? Did you initiate a clone and migrate? All storage will fail at some point, this storage failed during a maintenance window that was advertised two weeks out. I'm somewhat surprised they are able to declare defeat in less than 24 hours, I would expect moving the drives to another host would at least be able to recover the data, but I don't know how they designed their system, and some systems are more byzantine than others.
_nalply 6y ago
A few years ago I tried to cancel something, but it didn't work. Luckily my credit card was expiring anyway, so I just did nothing. But then they sent me emails more and more aggressive, so I put them on my killfile. Two years later I discovered that they were still sending me emails.
justinclift 6y ago
> Following extensive troubleshooting we have been unable to bring the host server on which your volume is hosted back online ... How is now being able to bring a server online connected to the storage that server provides? Surely they'd remove the physical storage devices from the dead server, put them in another one, and bring it up? ... unless they're hosting their servers with a third party they can't access? eg AWS or similar, which would be kind of ironic. ;)

134 points by sz4kerto 6y ago

wernerb 6y ago

This is incredibly frustrating, I can sympathize. But this is why we do backups even on cloud storage. I had the same thing happen with an EBS volume on AWS becoming unresponsive, unable to attach and load.

jtagen 6y ago

I'm surprised, always thought the Rackspace premium price was to prevent or significantly reduce these issues. That being said, I've had quite a few problems with them over the last year or so that is making me reconsider. The first-tier support they now outsource to India is frustrating. CloudFiles entries being "inaccessible" after saving for no known reason, happens a couple times a month. Had a managed MySQL slave replica down for 5 hours because they changed the IP of the master (mandatory maintenance) and failed to update the replica, no way to do this on my own. AWS and Azure might not be better, but they're definitely cheaper.

folkhack 6y ago
> I'm surprised, always thought the Rackspace premium price was to prevent or significantly reduce these issues. I've found that these systems are often there to make the business-minded customer feel better vs. offering up any sort of highly-competent technical support. Rackspace fits this bill for me although I haven't been their customer for close to 10 years due to how badly they've burnt me. On the contrary, free-tier Linode support has always been incredibly technically competent and has never let me down for the 10+ years I've been with them. If anyone from Linode is reading this - you guys are awesome, and your support staff is amazing. I can't say enough nice things about them as they've done so much to support me for a solid decade! All I'm saying is paying for support vs. not paying doesn't mean you'll have any more technically competent help in my real-world experience. It just means "you're paying for support" on paper, and typically is just an up-sell from legacy hosting providers like Rackspace or GoDaddy. Free from Linode is 100X paid from Rackspace - all depends on the company.
- joshribakoff 6y ago
  I moved a company off Linode a few years ago because of ongoing intermittent packet loss (think 93%) for subsets of our customers, they never were able to do anything about it and just pointed fingers at the ISPs, who of course I have no direct contact or relationship with. Wouldn’t recommend.
- snowwrestler 6y ago
  Rackspace support is not just “help you when you are having trouble,” they will do sysadmin tasks for you, basically as much as you want. It’s more like light professional services + typical support. That’s why it costs more.
  - folkhack 6y ago
    Yes, well aware. In-industry we call this "managed" and "un-managed" hosting. All of my issues actually happened with "managed" hosting 10 years ago. I was in my early-20's and still getting my feet underneath me as a Linux administrator so having a backup was something I advocated when I was provisioning our hosting. Frankly, it was probably pretty good for career development because their support staff was clueless with anything Linux administration, and if I needed an answer quick I was my only trustworthy resource. Like - basic LAMP (Linux, Apache2, MySQL, PHP) was completely beyond them even though that was the bread and butter of their business. I haven't sprung for "managed" hosting ever since because it just taught me that I was the best person to handle my interests... and it lit a fire under my butt to get way better at my Linux chops!
berns 6y ago
> AWS and Azure might not be better, but they're definitely cheaper. Azure has "ZERO % Annualized Failure Rate". It gives me such a cozy feeling.
- simonswords82 6y ago
  Amen. We moved to Azure after a DDOS attack which took out our Rackspace instances and we have never looked back.
simonswords82 6y ago
> AWS and Azure might not be better, but they're definitely cheaper. I can't speak for AWS as we don't use it but as a former client of Rackspace and a current client of Azure I can assure you that Azure is better and cheaper by a country mile.

kbar13 6y ago

yeah hardware issues happen, it's unfortunate but it happens on a very regular basis. backups are important

bartvk 6y ago
On one hand, I agree with you. On the other hand, you say this is a hardware issue but do you know for a fact that is what happened? Migration sounds to me like you migrate data from one state to the other state, then delete the old state. To me, it would be logical that before you delete the old state, you verify whether the migration was successful.
- kbar13 6y ago
  providers like this dont migrate data for no reason. there was probably a bad drive in the array or something, or some other known defect, so they are opting to migrate off the host before things get worse
tinus_hn 6y ago
The idea behind cloud services is that you pay someone to manage that.
- znpy 6y ago
  Yup it's ironic however how in the end you have to manage that stuff yourself anyway.
  - tetha 6y ago
    It depends how the vendor presents and handles it. I'm not aware how RS promises look. However, for example, the hetzner cloud has an explicit choice between triple-redundant ceph-based storage for VMs, or hypervisor local SSD storage. That's good and explicit. I can make an informed choice about speed or durability.
    - 6y ago

folkhack 6y ago

This is completely anecdotal: If you find yourself on Rackspace hosting I would absolutely migrate to something more mature. Personally I've had great luck with Digital Ocean and Linode for basic/affordable VPS hosting, with AWS being a great cloud provider with an incredible tool-box at your disposal. Hell, GPC, Azure... there's just so many better options. IMO Rackspace has been on a steady decline for years. They were a choice provider back in the early 2000's when a managed/unmanaged beige box/shared cPanel/Plesk hosting was common. Back then their support was incredible. Sometime around 2010 though things started to go down hill, and I experienced more and more support issues with them. These would range from annoying all the way to "that shouldn't happen" (like losing volumes, which happened to me). The final straw was when a dedicated server was turned down by mistake, causing a major outage for 20+ of my clients at the time. This was 10 years ago and I've never looked back. They did a pretty good job at turning me into an advocate against their services. Oh and the Slicehost transition was crap too. That sucked to get over to a first-gen VPS provider only to have it sucked right back into the beast that you were trying to escape from in the first place... Sorry for the mini rant. I'm still not over the PTSD that Rackspace has caused me.

ericcholis 6y ago
Currently, to delete a cloud server, you're required to call support. The option to delete servers via the API and interface have been removed. On more than one occasion I've been pitched by the "support engineer" to sign up for something akin to reserved instance pricing. They claim it's a security measure. I've never had an issue with service or availability. But, the price per performance on the VM instances is so much better elsewhere. Was working to migrate to DO, but now I'm fast tracking that process.
- folkhack 6y ago
  > Currently, to delete a cloud server, you're required to call support. This is insanity to me. I'd leave a VPS/cloud provider so fast if they started this. Even back in the day I always had the option to delete a Rackspace Cloud instance. Wow. Just wow.
- 6y ago

seneca 6y ago

Rackspace was sold to a private equity company in late 2016. Most of their senior staff has jumped ship since then, and services have been decreasing in quality. It's essentially a different company, and not one I'd host anything important on.

znpy 6y ago
Ha! I've been through something similar (company being sold). According to my experience, a company that passes from an owner to another one is basically another, completely different company.
x86_64Ubuntu 6y ago
Is there anyone that gets bought out by a PE company that doesn't go to absolute shit? QlikTech, Sencha, TravisCI all got bought and suffered a marked decline in service and roadmap confidence. Nowadays, when I hear someone is bought by a PE firm, I start looking for the exits.
- tedivm 6y ago
  I remember when LimeLight Networks got bought out by PE and their services turned to crap in about six months. It got so bad we got lawyers involved to break our contract with them.
- hn_throwaway_99 6y ago
  I really wonder about this when it comes to the longer term viability of tech-focused PE firms. The following seem to be absolutely truisms: 1. The second a company is bought out by PE, all the good employees run for the exits. Corporate America isn't exactly known for its empathy and compassion, but working for a PE-owned company means you can be sure the ownership gives 0 shits about anyone working at the company. I have never heard anything but horror stories of those working for PE-owned shops. 2. Given #1, it means the products of said companies always turn to shit pretty fast. This makes me really wonder about how PE firms can really work their financial magic in the long term. Perhaps they're just better at extracting value out of companies that are already declining in the first place, so they just suck the blood dry before they discard of the carcass?

rshnotsecure 6y ago

A few months ago I found that Rackspace had left open their prod logging system for their global load balancer / firewall system (known as BlueFlood). Reporting the open Elasticsearch server to then was one of the most bizarre experiences of my life. Ultimately the system was exposed for about 2 weeks. You can find some screenshots of what their backend looked like, which came in at about 3 TB daily, here: https://blog.12security.com/rackspace-bluefood-breach/ https://blog.12security.com/rackspace-bluefood-breach/

deleted 6y ago

[deleted]

QUFB 6y ago

Is this news? This is a block storage failure on Rackspace. EBS volumes on Amazon fail all the time, as anyone who manages a large number of instances probably knows (0.1%-0.2% per year for normal volumes): > Amazon EBS offers a higher durability volume (io2), that is designed to provide 99.999% durability with an annual failure rate (AFR) of 0.001%, where failure refers to a complete or partial loss of the volume. For example, if you have 100,000 EBS io2 volumes running for 1 year, you should expect only one io2 volume to experience a failure. This makes io2 ideal for business-critical applications such as SAP HANA, Oracle, Microsoft SQL Server and IBM DB2 that will benefit from higher uptime. io2 volumes are 2000 times more reliable than typical commodity disk drives, which fail with an AFR of around 2%. All other Amazon EBS volumes are designed to provided 99.8%-99.9% durability with an AFR of between 0.1% - 0.2%, https://aws.amazon.com/ebs/features/ https://aws.amazon.com/ebs/features/

deleted 6y ago
[deleted]

toast0 6y ago

> First, RS has informed us that they're migrating some volumes and that'll require downtime. I'm not trying to blame you, just trying to figure out what happened. Reading October 1st message, it didn't sound like they were going to clone and migrate it, but that they were letting you know something you could do to avoid the below work (which you didn't quote). What was the description of the work? Did you initiate a clone and migrate? All storage will fail at some point, this storage failed during a maintenance window that was advertised two weeks out. I'm somewhat surprised they are able to declare defeat in less than 24 hours, I would expect moving the drives to another host would at least be able to recover the data, but I don't know how they designed their system, and some systems are more byzantine than others.

_nalply 6y ago

A few years ago I tried to cancel something, but it didn't work. Luckily my credit card was expiring anyway, so I just did nothing. But then they sent me emails more and more aggressive, so I put them on my killfile. Two years later I discovered that they were still sending me emails.

justinclift 6y ago

> Following extensive troubleshooting we have been unable to bring the host server on which your volume is hosted back online ... How is now being able to bring a server online connected to the storage that server provides? Surely they'd remove the physical storage devices from the dead server, put them in another one, and bring it up? ... unless they're hosting their servers with a third party they can't access? eg AWS or similar, which would be kind of ironic. ;)