4 ms·

BuildYourVPS suffers massive data loss

You are receiving this email because you have a VPS hosted through TOCICI's BuildYourVPS services. We encountered a data loss issue on the storage array hosting your VPS environment, details of which follows. Your VPS was previously hosted from a hardware RAID based data storage array which encountered a firmware level fault earlier today. Our server vendor (IBM) had apparently not seen this particular bug prior to today, which has led to hours of non-stop troubleshooting and diagnostics throughout today (tracked at: http://status.tocici.com ). The effects of this issue are an inability to access any data stored on the inaccessible array: despite advanced troubleshooting alongside IBM engineers, we remain unable to access any data previously stored within your VPS. IBM is continuing to work with us, including a rapid shipment of replacement equipment, and in-depth root-cause evaluation assistance and support. Although in the meantime, in order to re-establish your VPS environment as quickly as possible, we have re-designated your VPS's host node; defining one of our existing standby nodes. You should see a new VPS build email soon. We share your frustration over this issue, especially with the sudden loss of multiple drives in a high-quality vendor's hardware based RAID...an incident that in nearly 20 years of doing business, we have never seen before. We are continuing to explore options, and we will keep you updated regarding any positive developments. Sincerely, TOCICI's Entire Support Team

MatthewPhillips 15y ago
I received this email at 3:26, my VPS was wiped out. They built new ones, and luckily I only had a hobby project static site sitting there. This is pure speculation but I feel like BuildYourVPS is running too many VMs on their servers. SSH has always felt sluggish. This is one of the downsides to a cheap host. I'd highly recommend prgmr.com (run by a fellow HNer), I've never had an issue with their VPSes at all.
- pestaa 15y ago
  From prgmr.com: ordering is disabled; I'm having some hardware issues I remember having seen this, and it was a long time ago.
  - MatthewPhillips 15y ago
    I consider this to be a feature, rather than adding new customers that they can't support fully, they turn off registration. Others might just stick new customers on the old hardware.
    - mootothemax 15y ago
      Another vote for prgmr - an incredible level of service for the price. And very much agreed on the feature angle ;)
    - pestaa 15y ago
      It might be a relieving feature for the insiders, but it unfortunately renders your post to "I highly recommend these closed doors".
      - MatthewPhillips 15y ago
        I don't think the doors are closed permanently. I know you said you have seen the message before but I think you've just checked at inconvenient times. I added a new VPS just a month ago. If you need a VPS tomorrow then yeah I can feel your pain.
jacknews 15y ago
It's one of the downsides to hardware raid. IMHO it is a liability, and obsolete.
- pestaa 15y ago
  What are the possible alternatives?
  - stl 15y ago
    ZFS would be a good alternative.
    - Ecio78 15y ago
      They said that they had a "firmware level fault", this could happen also on a card used by a ZFS storage, or you can have a bug in the ZFS code.. it would be nice to know which IBM storage they have 'cause I've just had a bug in the battery firmware of Netapp storage on Easter (IBM rebrands Netapp too)
jacknews 15y ago
It's one of the downsides to hardware raid. IMHO it is a liability, and obsolete.
losethos 15y ago
I'd rather be in jail than hell. God says... hot relators must tardy reputed think challenges perverse procured follow wizard disk comes joy enlargedst backward appeal glittering signs toward women ourself scourged broke Already an murdered embryo overturned End Surely doth whispers paradise celibacy messengers
omi 15y ago
While probably a different issue, a while back, IBM had messed up storage that's attached to an old mainframe. The nicest thing is that they didn't bother to tell us anything. It was only discovered as we started seeing FS corruption on a linux image.

MatthewPhillips 15y ago

I received this email at 3:26, my VPS was wiped out. They built new ones, and luckily I only had a hobby project static site sitting there. This is pure speculation but I feel like BuildYourVPS is running too many VMs on their servers. SSH has always felt sluggish. This is one of the downsides to a cheap host. I'd highly recommend prgmr.com (run by a fellow HNer), I've never had an issue with their VPSes at all.

pestaa 15y ago
From prgmr.com: ordering is disabled; I'm having some hardware issues I remember having seen this, and it was a long time ago.
- MatthewPhillips 15y ago
  I consider this to be a feature, rather than adding new customers that they can't support fully, they turn off registration. Others might just stick new customers on the old hardware.
  - mootothemax 15y ago
    Another vote for prgmr - an incredible level of service for the price. And very much agreed on the feature angle ;)
  - pestaa 15y ago
    It might be a relieving feature for the insiders, but it unfortunately renders your post to "I highly recommend these closed doors".
    - MatthewPhillips 15y ago
      I don't think the doors are closed permanently. I know you said you have seen the message before but I think you've just checked at inconvenient times. I added a new VPS just a month ago. If you need a VPS tomorrow then yeah I can feel your pain.

jacknews 15y ago

It's one of the downsides to hardware raid. IMHO it is a liability, and obsolete.

pestaa 15y ago
What are the possible alternatives?
- stl 15y ago
  ZFS would be a good alternative.
  - Ecio78 15y ago
    They said that they had a "firmware level fault", this could happen also on a card used by a ZFS storage, or you can have a bug in the ZFS code.. it would be nice to know which IBM storage they have 'cause I've just had a bug in the battery firmware of Netapp storage on Easter (IBM rebrands Netapp too)

jacknews 15y ago

It's one of the downsides to hardware raid. IMHO it is a liability, and obsolete.

losethos 15y ago

I'd rather be in jail than hell. God says... hot relators must tardy reputed think challenges perverse procured follow wizard disk comes joy enlargedst backward appeal glittering signs toward women ourself scourged broke Already an murdered embryo overturned End Surely doth whispers paradise celibacy messengers

omi 15y ago

While probably a different issue, a while back, IBM had messed up storage that's attached to an old mainframe. The nicest thing is that they didn't bother to tell us anything. It was only discovered as we started seeing FS corruption on a linux image.