6 ms·
Dear customer, This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an inciden
by webrobots 7y ago
Dear customer,
This mail is a follow-up to the previous email we sent (on January 8th,
2020) on this topic. As a reminder, yesterday, we experienced an
incident on a storage unit at our LU-BI1 datacenter, located in
Luxembourg.
Despite the replication systems in place, and the combined efforts of
our technical teams throughout the night, we were unable to reover the
data that was lost on the impacted storage unit.
We sincerely apologize for the inconvenience that this situation has
caused. This type of incident is extremely rare in the web hosting
industry.
In the event that you have a backup of your data, we suggest that you to
use it to recreate your server at a different datacenter.
To help you in this, we have provided you with a promo code that will
give you one free month for an instance, so that you can create a new
Simple Hosting instance in a different datacenter:
XXX
- petee 7y agoWow, for a company that boasts "no bullshit", only offering a month after destroying data and backups seems a little tone deaf Edit: in fairness, I'm not sure how exactly you would quantify such a loss anyway...
- Shalle135 7y agoIt sounds like they didn’t have any backups at all but rather relied on a active-active replication link to a secondary storage. Edit: who knows it may be related to the HPE issue. https://www.bleepingcomputer.com/news/hardware/hp-warns-that-some-ssd-drives-will-fail-at-32-768-hours-of-use/ https://www.bleepingcomputer.com/news/hardware/hp-warns-that...
- jnwatson 7y agoIn other words, RAID is not backup.
- mschuster91 7y agoWhat baffles me is that there seems to be no way for either the customer or a data-recovery company to flash a new firmware onto the drive after it has failed. Someone there wanted to spare the few millicents of copper trace for a JTAG port?!
- marcinzm 7y agoProbably to prevent supply chain firmware changes for hacking, espionage, etc.
- deelowe 7y agoHmm... I wonder what the "incident" was. If it involved something akin to an "rm -rf," then of course their replication link didn't protect them.
- fencepost 7y agoPerhaps they were depending on snapshotting and were not prepared for some kind of hardware failure taking out the entire storage system.
- joepie91_ 7y agoReputable hosting providers typically don't try to quantify such a loss, but rather outright offer a credit/compensation that is very obviously generous (say, a year or even two of free service). Especially when a small set of your customerbase is affected, it won't cost you that much, and "overcompensating" like that means that virtually noone is going to criticize you for quantifying it wrong; instead, the public narrative will be centered around "well, shit happens, they did their best and generously compensated".
- mmoez 7y agoA promo code in exchange of your data loss. What a bargain!
- atlasAUPrivacy 7y ago“Please keep trusting us to host your data”
- scohesc 7y agoYou really shouldn't trust anyone hosting your data. Always have backups!
- oefrha 7y agoOften times the backup provider is the hosting provider, whom you have to trust. (This extends all the way from big clouds like AWS and GCE to small providers like Linode and DO). Having an external backup can be unreasonably expensive due to ridiculous egress costs.
- arpa 7y agoYou can still back up to the same providers' different data center. Two data centers failing simultaneously is very unlikely.
- oefrha 7y agoNot always an option. For instance, I use Linode’s backup service and it can only back up to the same data center (although it is said to live on a separate system).
- jdmguit 7y agoComing from a Linode employee, I can confirm this is true. Linode's backups live in the same data center as the server, but the systems are separated so that they don't directly affect one another.
- hedora 7y agoInteresting. The public status page says they’re still waiting for the recovery process to complete.
- manuelmagic 7y agoI could understand the incident (I would _at least_ start questioning myself about the quality of the service I'm paying), but IMHO this is not something that can be addressed with a casual e-mail that contains few lines of excuses and a "promo code" like it's everyday business. That's astonishing. Worse than a bad incident there is only bad management of the following situation.
- abtinf 7y ago> This type of incident is extremely rare in the web hosting industry. Why would they include that sentence? Are they trying to imply it is rare for them because it is rare for the industry? Are they saying they are not as good as the industry, so customers should move to other providers? Or are they trying to show they apply the same inattention to their customer communication as they apply to their data backup/recovery practices? This kind of data loss should simply never happen. It’s one thing to say “it will take us up to 30 days to restore your data because our fast recovery options aren’t working and we have to bring up cold archives”, it’s entirely another to say “your data is gone, tough”.
- SmellyGeekBoy 7y agoI'm not sure why you've been downvoted for this. I thought the same. I read it as: "This type of incident is extremely rare in the web hosting industry, because apparently the overwhelming majority of our competitors aren't capable of fucking up as badly as we just did." Doesn't inspire confidence at all, IMO.
- ben509 7y ago> Why would they include that sentence? They're a French company; it may be a non-native speaker not catching the implication. It's also possibly an editing error, e.g. they started writing something like, "these types of incidents are extremely rare and when they happen etc" and most of it was dropped without considering how that changed the implication.
- jermops 7y agoIf you're not paying for backups... what archive?
- icebraining 7y agoThey say you can backup by using their snapshoting tool, but they lost those snapshots too.
- yjftsjthsd-h 7y ago
- passivepinetree 7y agoIs this a response from the company or are you putting it forth as an example response for how to handle this incident better? It’s unclear from your post.
- jandrese 7y ago> This type of incident is extremely rare in the web hosting industry. I read this as "so maybe you should consider one of the other web hosting companies that doesn't have problems like this."