5 ms·

Microsoft/Danger lose Sidekick user data in server failure

60 points by bensummers 17y ago

lallysingh 17y ago
This is really, really unacceptable. I don't know how you screw something up this badly. HINT: http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&Description=tape+drive http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&#3... Additionally, I suspect we're going to see these customers really hesitate on future cloud-based services.
- eli 17y ago
  I'm guessing they have a lot more data than is practical to store on a tape.
  - rbanffy 17y ago
    You do it by having redundant storage. I am admittedly not a good sysadmin, but I keep an eye on my own "pirate" server (that is, outside the realm of the corporate network) and I have already ordered additional storage because Munin shows that, at present consumption rates, we will run out of storage in about 4 months, crossing the 70% mark in three. I know they will take at about two months for corpotate IT to spend the US$ 100 it costs for a pair of disk-drives and that gives me a month to set-up a soft RAID before I cross the 70% mark. It's completely unacceptable not to have the storage mirrored and load-balanced on a second SAN. And mind you many SAN vendors even bundle software that makes it easy to do. Even if you don't have the budget to buy another SAN the same size, you can mirror the data rather easily with a bunch of servers, each with a bunch of SATA cards and a bunch of low-budget drives. Redundancy is the key here - if you have a drive that's 10x cheaper than the same storage in the SAN, you can replicate in a RAID1 with 10 drives. If each drive is 30% as reliable as the high-grade disks that go into the SAN, you are well on the winning side. And I seriously doubt there is that much data. How much? A couple dozen terabytes? That's fits easily under my desk. And quite probably in my discretionary budget.
- wrinklz 17y ago
  Perhaps scaring people from "the cloud" was part of the Borg's plan...
joevandyk 17y ago
Yikes. I assume they didn't have offsite backups, and some dude did a 'rm -rf *' or ran a 'delete from table cascade' command.
- 4buot 17y ago
  Apparently they botched an upgrade of their Oracle SAN.
- nkassis 17y ago
  I mean this could count for criminal negligence. I wish they would release more info on how the failure happened and why the backups aren't working.
- eli 17y ago
  Seems like a big assumption since we don't know any details. Perhaps a malicious sysadmin?
- jsz0 17y ago
  Based on the wording of the statement it seems likely they had no usable backup solution at all. Offsite or otherwise. Even a 24 hour daily backup should have provided some data. Sounds like a classic case of neglect to me. They probably had a solution in place but didn't verify it was actually working.
jsz0 17y ago
The timing of this is a bit suspicious. I just read an article late last week from an "insider" detailing what a major clusterfuck the Danger acquisition has been for Microsoft and how they've completely neglected the SideKick platform in favor of their own failed attempts to create a competing device. The speculation from this insider being that Microsoft had zero interest in the NetBSD based SideKick and only bought Danger to kill the SideKick and launch their own product to replace it. Most of the original developers are long gone so that makes me wonder who was taking care of the SideKick backend servers. Are they perhaps UNIX based servers? Did Microsoft perhaps throw a bunch of Windows Server guys at them? (that never ends well)
- nkassis 17y ago
  I don't know, why did they buy them in the first place? Just to kill them? Did they fire the whole system staff they used to have? I just can't believe they just know stopped running backups and let it die. Sounds odd, and prone to being sued by T-Mobile quick. (Maybe Microsoft is going after T-Mobile for android?)
  - jsz0 17y ago
    http://www.appleinsider.com/articles/09/10/09/exclusive_pink_danger_leaks_from_microsofts_windows_phone.html http://www.appleinsider.com/articles/09/10/09/exclusive_pink... The most relevant bits: "The final operator who is going to be pissed is T-Mobile, who has been just as loyal of a partner to Danger as Sharp has been. I don't know exactly what Microsoft has been telling them, but they have no doubt realized that they've been cut out of this deal in favor of their largest competitor. What's worse is that apparently Microsoft has been lying to them this whole time about the amount of resources that they've been putting behind Sidekick development and support (in reality, it was cut down to a handful of people in Palo Alto managing some contractors in Romania, Ukraine, etc.). "The reason for the deceit wasn't purely to cover up the development of Pink but also because Microsoft could get more money from T-Mobile for their support contract if T-Mobile thought that there were still hundreds of engineers working on the Sidekick platform. As we saw from their recent embarrassment with Sidekick data outages, that has clearly not been the case for some time."
SamAtt 17y ago
This probably (definitely?) makes me a jerk but I have to say it: They deserved it. I'm sorry but in this day and age anyone trusting their data to a device that stores it exclusively on the server deserves what they get. Some will say "the type of person who uses these devices isn't the type of person to understand where their data is located" but I say that's BS. We're in a data centric world now and I don't think it's asking that much for people to be aware of where their data is kept and to make sure they have some kind of "backup" guarantee. To Anyone Downvoting This: Let me ask you one question. If someone without a seatbelt gets hit by a drunk driver does the fact that the accident's the drunk driver's fault mean it was ok for them not to wear a seatbelt? Or can someone be responsible for their own irresponsibility even though the damage was caused by a party that was more to blame?
- systems 17y ago
  If you use Gmail, Facebook, or any kind of SaaS, you are very much from "this day and age" and using the latest from the greatest and is trusting your data exclusively to a server! Heck, you trust your money to a bank! You trust your bus driver not to crash the bus, your doctor to properly heal you, and your lawyer to put you out of trouble, you also trust your cook to make the meal you like. Everyday, you give trust to many individuals and entities to whom you delegate tasks that you may or may not know how to do. Sidekick screwed up, but I would never blame the ones who trusted them what they(the clients) did is completely normal!
  - SamAtt 17y ago
    I'm sorry but if you are content to leave the only copy of priceless things such as contacts, treasured photos, etc... in the hands of companies with no contractual obligation to you than yes, you're a fool. The bank legally owes me my money. The bus driver is guilty of a crime if he crashes the bus. The Doctor is guilty of malpractice. If gmail deletes your account you have absolutely not recourse what so ever. None, Zip. Zilch. Zero. Sidekick did screw up but it doesn't make the people who lost their data innocent either. You have to take responsibility for that which you value.
- JimmyL 17y ago
andr 17y ago
This is a good reason to think of what are the quickest ways to bring one of your own sites down and try to prevent them. In 2002 I lost the data to a small (~200 users) hobby site I built and to this date it still haunts me as my biggest failure. So now I'm taking precautions such as offsite backups, forbidding DELETE for all SQL users, etc., to limit the chance of such a thing happening.
megaduck 17y ago
Oh. My. God. I was one of the first Sidekick customers, long before the MS buyout. It was an amazing email/texting device, far easier and more comfortable to use than a Blackberry. Plus the web browsing was decent, and the UI was simple and elegant. All in all, a really cool phone. Unfortunately, the first generation of hardware was super flaky. I had five of them die on me in a year, often failing within weeks. Since there wasn't any PC sync tool, the only thing that saved my bacon was the server-side data model. When a phone died, I'd get a replacement, drop in the SIM card and all my stuff would be there same as always. Back in the day, there was no data sync with your PC. I think they eventually released Outlook syncronization, but most consumers don't use Exchange/Outlook. I would guarantee that a huge number of Sidekick owners (most? all?) have never had any of their phone data anywhere but the Danger servers. To make matters worse, the Sidekick becomes kind of a brick when the servers are down. I can see a lot of people resetting their device because it's suddenly become unresponsive, just expecting their data to magically come back from the server. This is an unmitigated disaster.
- covercash 17y ago
  I was a loyal Sidekick user right up until the day the first gen iPhone launched. It really was a fantastic device with the always on AIM, server side re-rendering of websites to optimize them for the device and one of the best physical keyboards I have used to this date. If I remember correctly, Mark Cuban was a sidekick fanatic. I wonder what his reaction to this debacle is.
  - white_eskimo 17y ago
    Server side re-rendering of websites?? Sounds like wide-spread support for that would be lacking... I'm sure as the Sidekick gained traction, more web sites adopted its special rendering requirements as you mention, but in the beginning, things must have been rough! Disclosure - I've never owned a Sidekick (and know little about them, other than they are apart of Microsoft's 3 screens and a cloud strategy, yet they don't run Windows Mobile (??)).
    - 17y ago
chrischen 17y ago
The dangers of the cloud... Edit: Long live local copies!
- jonursenbach 17y ago
  I don't think this is really showcasing the dangers of the cloud, but rather the dangers of companies not knowing what the fuck they're doing.
  - btn 17y ago
    When everything is in the 'cloud', how do you know the difference?
  - MattCutts 17y ago
    Or the dangers of companies not allowing you to easily export/backup your data.
    - gry 17y ago
      I'm not sure how much is a tech feature or un-walled policy would solve the problem. I think it would fractionally minimize it. The current psychological barrier -- people not using a phone as a computer and its imperfections is profound. Before, if a phone didn't work, it was one of a few things, usually the device or the carrier. Now, it's one of a dozen dozen or more products and services. People couldn't export before. They might be able to now. A friend of mine once noted the subtle difference between how technology can protect you from failure, but a backup can help protect you from yourself. It feels like we're moving towards a singularity. Expectations, responsibility and technology.
    - Elepsis 17y ago
      While this is true in principle, this is a bit of a Google talking point that has more PR cachet than practical implications. Giving users the ability to make backups of their data certainly does not mean they will actually do so. How many Gmail users actually keep full backups of their email (using POP or IMAP or whatever)? And if Gmail had a case of massive data loss tomorrow, do you really think saying "well, you could've made a backup just fine" would have somehow made everything okay? I don't deny that this sort of data loss is pretty awful, but "data liberation" is not the solution.
johnnybgoode 17y ago
This is incredible. In no way is it even remotely acceptable for them not to have backups.
- viraptor 17y ago
  It doesn't have to be the case of no backups. Many different things can happen... They could have a faulty backup design. They could have lost the backups. They could have a fault in their backup verification / checking procedure. They could miss some part of the data that wasn't crucial before, but after a couple of updates, they cannot restore contacts without it. Etc. There are many ways this could have happened - not that they are excused because of that, but it's not fair to say "not to have backups" until we know the whole story.
  - ams6110 17y ago
    Backups are only as good as your ability to recover from them. How many of us think we are doing a good job backing up data, AND have actually tried to run a disaster recovery drill and have verified that you can recover your systems from your backups? You don't want to find the gaps in you backup strategy during a real system failure, when you're under the pressure of the clock to get the system back up.
  - johnnybgoode 17y ago
    As far as I'm concerned, if you can't recover your data, you don't really have a backup. That covers every one of your hypotheticals.
    - viraptor 17y ago
      But you cannot be sure you can recover your data (you can be sure in theory only). In practice, you don't know if you can recover your data unless you migrate the whole company to the backup copy and ask them to check every functionality they use -- every time you make a copy (and rely on them doing 100% coverage). The best you can have during the normal operation is a high probability that what you get is a proper backup, because that would be the case the last time you did a full test (if you ever did). Datasets change, ways of interfacing with data change, etc. Testing is not perfect either, because your tests can be broken or can be reporting false positives. So yeah... noone can be sure they can recover the data. Does that mean that noone really has a backup?
natch 17y ago
"We recognize the magnitude of this inconvenience." Inconvenience. Wow. Catastrophe seems to fit better.
- fuzzmeister 17y ago
  If it's something that cannot be fixed (these customers are never going to get their text messages back, for example), it is no longer an inconvenience.
- tptacek 17y ago
  They used the word "magnitude", which seems like the closest they could probably come to "clusterfuck" without the wording itself being legally actionable.
Sidnicious 17y ago
"We continue to advise customers to NOT reset their device by removing the battery or letting their battery drain completely, as any personal content that currently resides on your device will be lost." This is the most shocking part of the article. Is the system really set up so that a failure on the server side causes data on the devices to be deleted? That strikes me as bad design. Yes, any sync system comes with a built-in danger that one endpoint will unexpectedly delete data on another, but if servers goes down the phone should hold onto its data until told otherwise.
- drusenko 17y ago
  I read it as more of a cache -- the phone caches data that you access, and that cache is reset when the phone is rebooted.
Dilpil 17y ago
We don't need offsite backup. Server crashes are something that happens to other companies.
hristov 17y ago
Snoop will be pissed.
aschobel 17y ago
A plausible rumor is that the data loss stems from a botched SAN upgrade by Hitachi: Currently the rumor with the most weight is as follows: Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in it’s destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved. http://www.hiptop3.com/archives/what-caused-the-sidekick-fail http://www.hiptop3.com/archives/what-caused-the-sidekick-fai...
- eli 17y ago
  If you've got a really big SAN, it's pretty tough to back it up completely and independently without... a second really big SAN
  - tptacek 17y ago
    I've seen people "back up" NetApp SANs before to prepare for upgrades; I don't think it's generally accepted practice to just say "fuck it, we'll do it live!" They don't use "big tapes". They plan a disaster recovery architecture and (in the cases I've been involved with) use block streaming protocols to mirror data. Granted, I didn't do the backups, but I'm pretty sure they didn't pay us to bust up those streaming protocols so they could not use them.
gry 17y ago
At one of my haunts, I oft talk to a seasoned and grizzled developer. He's convinced everyone, and I mean everyone, should know how the internet works. My mom should know the OSI model, he argues. I suggest it's unreasonable. As a cell phone user to my provider, my relationship and contract starts and ends there. Then, it gets complex. And yet, I expect it to work without fail. It seems like it infrastructures are credit default swap complex to the average consumer. This is a bad thing. I don't expect an people to should understand the OSI model. I fully expect we'll find a balance between people trusting others with their information (features/convenience) and having it mirrored/synched without disaster locally (nice to be home).
brianobush 17y ago
The best way to prevent data loss is to put your data on a public server and have google index it. Google's cache is awesome and yes I have used this before :)
- yosh 17y ago
  "Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it." -- Linus Torvalds
mbrubeck 17y ago
This is just one of the reasons I use http://lifestreambackup.com/ http://lifestreambackup.com/ to make my own back ups of data in online services.
rbanffy 17y ago
I smell a migration of their infrastructure to Windows and a nice "Get the Facts" page about this disaster, how it could be blamed on non-Windows technology and how everything was saved (or, at least, made safe) by the deployment of Windows Server 2008.
- dtf 17y ago
  Word on the street is that it was a botched SAN upgrade by a Hitachi tech. Of course, the fact that the Danger admins hadn't backed up the SAN before the upgrade made that fatal. Not even an offsite backup... crazy.
  - rbanffy 17y ago
    And that would be one of the bullet-points in the page: how hard it is to properly back-up stuff in *BSD as compared to how easy and foolproof it is under Windows 2008 Server. In fact, I can almost smell a migration directly to Windows 7 Server or whatever the name they decide to use for that.

60 points by bensummers 17y ago

This is really, really unacceptable. I don't know how you screw something up this badly. HINT: http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&Description=tape+drive http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&#3... Additionally, I suspect we're going to see these customers really hesitate on future cloud-based services.

eli 17y ago
I'm guessing they have a lot more data than is practical to store on a tape.
- rbanffy 17y ago
  You do it by having redundant storage. I am admittedly not a good sysadmin, but I keep an eye on my own "pirate" server (that is, outside the realm of the corporate network) and I have already ordered additional storage because Munin shows that, at present consumption rates, we will run out of storage in about 4 months, crossing the 70% mark in three. I know they will take at about two months for corpotate IT to spend the US$ 100 it costs for a pair of disk-drives and that gives me a month to set-up a soft RAID before I cross the 70% mark. It's completely unacceptable not to have the storage mirrored and load-balanced on a second SAN. And mind you many SAN vendors even bundle software that makes it easy to do. Even if you don't have the budget to buy another SAN the same size, you can mirror the data rather easily with a bunch of servers, each with a bunch of SATA cards and a bunch of low-budget drives. Redundancy is the key here - if you have a drive that's 10x cheaper than the same storage in the SAN, you can replicate in a RAID1 with 10 drives. If each drive is 30% as reliable as the high-grade disks that go into the SAN, you are well on the winning side. And I seriously doubt there is that much data. How much? A couple dozen terabytes? That's fits easily under my desk. And quite probably in my discretionary budget.
wrinklz 17y ago
Perhaps scaring people from "the cloud" was part of the Borg's plan...

joevandyk 17y ago

Yikes. I assume they didn't have offsite backups, and some dude did a 'rm -rf *' or ran a 'delete from table cascade' command.

4buot 17y ago
Apparently they botched an upgrade of their Oracle SAN.
nkassis 17y ago
I mean this could count for criminal negligence. I wish they would release more info on how the failure happened and why the backups aren't working.
eli 17y ago
Seems like a big assumption since we don't know any details. Perhaps a malicious sysadmin?
jsz0 17y ago
Based on the wording of the statement it seems likely they had no usable backup solution at all. Offsite or otherwise. Even a 24 hour daily backup should have provided some data. Sounds like a classic case of neglect to me. They probably had a solution in place but didn't verify it was actually working.

jsz0 17y ago

The timing of this is a bit suspicious. I just read an article late last week from an "insider" detailing what a major clusterfuck the Danger acquisition has been for Microsoft and how they've completely neglected the SideKick platform in favor of their own failed attempts to create a competing device. The speculation from this insider being that Microsoft had zero interest in the NetBSD based SideKick and only bought Danger to kill the SideKick and launch their own product to replace it. Most of the original developers are long gone so that makes me wonder who was taking care of the SideKick backend servers. Are they perhaps UNIX based servers? Did Microsoft perhaps throw a bunch of Windows Server guys at them? (that never ends well)

nkassis 17y ago
I don't know, why did they buy them in the first place? Just to kill them? Did they fire the whole system staff they used to have? I just can't believe they just know stopped running backups and let it die. Sounds odd, and prone to being sued by T-Mobile quick. (Maybe Microsoft is going after T-Mobile for android?)
- jsz0 17y ago
  http://www.appleinsider.com/articles/09/10/09/exclusive_pink_danger_leaks_from_microsofts_windows_phone.html http://www.appleinsider.com/articles/09/10/09/exclusive_pink... The most relevant bits: "The final operator who is going to be pissed is T-Mobile, who has been just as loyal of a partner to Danger as Sharp has been. I don't know exactly what Microsoft has been telling them, but they have no doubt realized that they've been cut out of this deal in favor of their largest competitor. What's worse is that apparently Microsoft has been lying to them this whole time about the amount of resources that they've been putting behind Sidekick development and support (in reality, it was cut down to a handful of people in Palo Alto managing some contractors in Romania, Ukraine, etc.). "The reason for the deceit wasn't purely to cover up the development of Pink but also because Microsoft could get more money from T-Mobile for their support contract if T-Mobile thought that there were still hundreds of engineers working on the Sidekick platform. As we saw from their recent embarrassment with Sidekick data outages, that has clearly not been the case for some time."

SamAtt 17y ago

This probably (definitely?) makes me a jerk but I have to say it: They deserved it. I'm sorry but in this day and age anyone trusting their data to a device that stores it exclusively on the server deserves what they get. Some will say "the type of person who uses these devices isn't the type of person to understand where their data is located" but I say that's BS. We're in a data centric world now and I don't think it's asking that much for people to be aware of where their data is kept and to make sure they have some kind of "backup" guarantee. To Anyone Downvoting This: Let me ask you one question. If someone without a seatbelt gets hit by a drunk driver does the fact that the accident's the drunk driver's fault mean it was ok for them not to wear a seatbelt? Or can someone be responsible for their own irresponsibility even though the damage was caused by a party that was more to blame?

systems 17y ago
If you use Gmail, Facebook, or any kind of SaaS, you are very much from "this day and age" and using the latest from the greatest and is trusting your data exclusively to a server! Heck, you trust your money to a bank! You trust your bus driver not to crash the bus, your doctor to properly heal you, and your lawyer to put you out of trouble, you also trust your cook to make the meal you like. Everyday, you give trust to many individuals and entities to whom you delegate tasks that you may or may not know how to do. Sidekick screwed up, but I would never blame the ones who trusted them what they(the clients) did is completely normal!
- SamAtt 17y ago
  I'm sorry but if you are content to leave the only copy of priceless things such as contacts, treasured photos, etc... in the hands of companies with no contractual obligation to you than yes, you're a fool. The bank legally owes me my money. The bus driver is guilty of a crime if he crashes the bus. The Doctor is guilty of malpractice. If gmail deletes your account you have absolutely not recourse what so ever. None, Zip. Zilch. Zero. Sidekick did screw up but it doesn't make the people who lost their data innocent either. You have to take responsibility for that which you value.
JimmyL 17y ago

andr 17y ago

This is a good reason to think of what are the quickest ways to bring one of your own sites down and try to prevent them. In 2002 I lost the data to a small (~200 users) hobby site I built and to this date it still haunts me as my biggest failure. So now I'm taking precautions such as offsite backups, forbidding DELETE for all SQL users, etc., to limit the chance of such a thing happening.

megaduck 17y ago

Oh. My. God. I was one of the first Sidekick customers, long before the MS buyout. It was an amazing email/texting device, far easier and more comfortable to use than a Blackberry. Plus the web browsing was decent, and the UI was simple and elegant. All in all, a really cool phone. Unfortunately, the first generation of hardware was super flaky. I had five of them die on me in a year, often failing within weeks. Since there wasn't any PC sync tool, the only thing that saved my bacon was the server-side data model. When a phone died, I'd get a replacement, drop in the SIM card and all my stuff would be there same as always. Back in the day, there was no data sync with your PC. I think they eventually released Outlook syncronization, but most consumers don't use Exchange/Outlook. I would guarantee that a huge number of Sidekick owners (most? all?) have never had any of their phone data anywhere but the Danger servers. To make matters worse, the Sidekick becomes kind of a brick when the servers are down. I can see a lot of people resetting their device because it's suddenly become unresponsive, just expecting their data to magically come back from the server. This is an unmitigated disaster.

covercash 17y ago
I was a loyal Sidekick user right up until the day the first gen iPhone launched. It really was a fantastic device with the always on AIM, server side re-rendering of websites to optimize them for the device and one of the best physical keyboards I have used to this date. If I remember correctly, Mark Cuban was a sidekick fanatic. I wonder what his reaction to this debacle is.
- white_eskimo 17y ago
  Server side re-rendering of websites?? Sounds like wide-spread support for that would be lacking... I'm sure as the Sidekick gained traction, more web sites adopted its special rendering requirements as you mention, but in the beginning, things must have been rough! Disclosure - I've never owned a Sidekick (and know little about them, other than they are apart of Microsoft's 3 screens and a cloud strategy, yet they don't run Windows Mobile (??)).
  - 17y ago

chrischen 17y ago

The dangers of the cloud... Edit: Long live local copies!

jonursenbach 17y ago
I don't think this is really showcasing the dangers of the cloud, but rather the dangers of companies not knowing what the fuck they're doing.
- btn 17y ago
  When everything is in the 'cloud', how do you know the difference?
- MattCutts 17y ago
  Or the dangers of companies not allowing you to easily export/backup your data.
  - gry 17y ago
    I'm not sure how much is a tech feature or un-walled policy would solve the problem. I think it would fractionally minimize it. The current psychological barrier -- people not using a phone as a computer and its imperfections is profound. Before, if a phone didn't work, it was one of a few things, usually the device or the carrier. Now, it's one of a dozen dozen or more products and services. People couldn't export before. They might be able to now. A friend of mine once noted the subtle difference between how technology can protect you from failure, but a backup can help protect you from yourself. It feels like we're moving towards a singularity. Expectations, responsibility and technology.
  - Elepsis 17y ago
    While this is true in principle, this is a bit of a Google talking point that has more PR cachet than practical implications. Giving users the ability to make backups of their data certainly does not mean they will actually do so. How many Gmail users actually keep full backups of their email (using POP or IMAP or whatever)? And if Gmail had a case of massive data loss tomorrow, do you really think saying "well, you could've made a backup just fine" would have somehow made everything okay? I don't deny that this sort of data loss is pretty awful, but "data liberation" is not the solution.

johnnybgoode 17y ago

This is incredible. In no way is it even remotely acceptable for them not to have backups.

viraptor 17y ago
It doesn't have to be the case of no backups. Many different things can happen... They could have a faulty backup design. They could have lost the backups. They could have a fault in their backup verification / checking procedure. They could miss some part of the data that wasn't crucial before, but after a couple of updates, they cannot restore contacts without it. Etc. There are many ways this could have happened - not that they are excused because of that, but it's not fair to say "not to have backups" until we know the whole story.
- ams6110 17y ago
  Backups are only as good as your ability to recover from them. How many of us think we are doing a good job backing up data, AND have actually tried to run a disaster recovery drill and have verified that you can recover your systems from your backups? You don't want to find the gaps in you backup strategy during a real system failure, when you're under the pressure of the clock to get the system back up.
- johnnybgoode 17y ago
  As far as I'm concerned, if you can't recover your data, you don't really have a backup. That covers every one of your hypotheticals.
  - viraptor 17y ago
    But you cannot be sure you can recover your data (you can be sure in theory only). In practice, you don't know if you can recover your data unless you migrate the whole company to the backup copy and ask them to check every functionality they use -- every time you make a copy (and rely on them doing 100% coverage). The best you can have during the normal operation is a high probability that what you get is a proper backup, because that would be the case the last time you did a full test (if you ever did). Datasets change, ways of interfacing with data change, etc. Testing is not perfect either, because your tests can be broken or can be reporting false positives. So yeah... noone can be sure they can recover the data. Does that mean that noone really has a backup?

natch 17y ago

"We recognize the magnitude of this inconvenience." Inconvenience. Wow. Catastrophe seems to fit better.

fuzzmeister 17y ago
If it's something that cannot be fixed (these customers are never going to get their text messages back, for example), it is no longer an inconvenience.
tptacek 17y ago
They used the word "magnitude", which seems like the closest they could probably come to "clusterfuck" without the wording itself being legally actionable.

Sidnicious 17y ago

"We continue to advise customers to NOT reset their device by removing the battery or letting their battery drain completely, as any personal content that currently resides on your device will be lost." This is the most shocking part of the article. Is the system really set up so that a failure on the server side causes data on the devices to be deleted? That strikes me as bad design. Yes, any sync system comes with a built-in danger that one endpoint will unexpectedly delete data on another, but if servers goes down the phone should hold onto its data until told otherwise.

drusenko 17y ago
I read it as more of a cache -- the phone caches data that you access, and that cache is reset when the phone is rebooted.

Dilpil 17y ago

We don't need offsite backup. Server crashes are something that happens to other companies.

hristov 17y ago

Snoop will be pissed.

aschobel 17y ago

A plausible rumor is that the data loss stems from a botched SAN upgrade by Hitachi: Currently the rumor with the most weight is as follows: Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in it’s destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved. http://www.hiptop3.com/archives/what-caused-the-sidekick-fail http://www.hiptop3.com/archives/what-caused-the-sidekick-fai...

eli 17y ago
If you've got a really big SAN, it's pretty tough to back it up completely and independently without... a second really big SAN
- tptacek 17y ago
  I've seen people "back up" NetApp SANs before to prepare for upgrades; I don't think it's generally accepted practice to just say "fuck it, we'll do it live!" They don't use "big tapes". They plan a disaster recovery architecture and (in the cases I've been involved with) use block streaming protocols to mirror data. Granted, I didn't do the backups, but I'm pretty sure they didn't pay us to bust up those streaming protocols so they could not use them.

gry 17y ago

At one of my haunts, I oft talk to a seasoned and grizzled developer. He's convinced everyone, and I mean everyone, should know how the internet works. My mom should know the OSI model, he argues. I suggest it's unreasonable. As a cell phone user to my provider, my relationship and contract starts and ends there. Then, it gets complex. And yet, I expect it to work without fail. It seems like it infrastructures are credit default swap complex to the average consumer. This is a bad thing. I don't expect an people to should understand the OSI model. I fully expect we'll find a balance between people trusting others with their information (features/convenience) and having it mirrored/synched without disaster locally (nice to be home).

brianobush 17y ago

The best way to prevent data loss is to put your data on a public server and have google index it. Google's cache is awesome and yes I have used this before :)

yosh 17y ago
"Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it." -- Linus Torvalds

mbrubeck 17y ago

This is just one of the reasons I use http://lifestreambackup.com/ http://lifestreambackup.com/ to make my own back ups of data in online services.

rbanffy 17y ago

I smell a migration of their infrastructure to Windows and a nice "Get the Facts" page about this disaster, how it could be blamed on non-Windows technology and how everything was saved (or, at least, made safe) by the deployment of Windows Server 2008.

dtf 17y ago
Word on the street is that it was a botched SAN upgrade by a Hitachi tech. Of course, the fact that the Danger admins hadn't backed up the SAN before the upgrade made that fatal. Not even an offsite backup... crazy.
- rbanffy 17y ago
  And that would be one of the bullet-points in the page: how hard it is to properly back-up stuff in *BSD as compared to how easy and foolproof it is under Windows 2008 Server. In fact, I can almost smell a migration directly to Windows 7 Server or whatever the name they decide to use for that.