5 ms·

Microsoft says it had Sidekick data backup, but it was hit, too

15 points by chuck_taylor 17y ago

jacquesm 17y ago
I find it almost unbelievable that there were no cyclic backups.
viraptor 17y ago
> Will this prompt Microsoft to review its policy for data backups? "This was an extraordinary situation [...]. We greatly apologize [...]. We have a full team working in parallel with the restoration team to augment the redundancy and stability of this acquired service over the long-term." That's got to be the best non-answer answer I've read in a long time. And no - redundancy and stability of the service have nothing to do with the backup policy.
- dlsspy 17y ago
  You forgot "acquired." It was them. Those bastards we bought. I mean, sure that was almost a year ago, and most of the original employees are gone, but do keep in mind when we're talking about this that it's not a proper MS offering. Unless you like it. Then yay us.
brown9-2 17y ago
Wish this article had more details than just "the backups were impacted too". A few months ago Google had a large multi-hour App Engine outage. Afterwards (within 24 hours?) they posted a detailed technical explanation/postmortem online for anyone interested to read, with a play-by-play that even went minute-by-minute. Microsoft/Danger owe their customers more of a real explanation.
InclinedPlane 17y ago
Wrong, wrong, wrong. If you have a backup that CAN be impacted by a problem in the main server then you do not have a backup at all, you have, at best, redundancy. The very reason to have a legitimate backup is that it cannot be damaged by the same mechanisms that can damage your primary data. It should be inert, if possible, or at least to some extent immutable, and it should be offsite. RAID is not a backup. SAN is not a backup. Mirroring to a redundant storage array is also not a true backup. Backups are hard to get right, and expensive. But the cost of getting it wrong can be monumental.
- andrewvc 17y ago
  I couldn't agree more. Real backups mean that at some level you can't delete that data, no matter how much you want to, without driving out to some secure remote location and demagnetizing some tapes.
  - furyg3 17y ago
    Real backups mean that at some level you can't delete that data It goes even farther than that: Real backups mean that at some level you can recover that data
makecheck 17y ago
In my experience, new "policies" put in place in these situations don't really do anything. They just inject bureaucrats where there were none before, and aggravate the few people remaining who might really help. And you can bet managers will start a witch hunt, firing some mostly-competent guy as a scapegoat (which may "feel good" at the time, but ironically will only slow down their recovery and set them up for their next failure). The smart thing would be for Microsoft to think of the employees: what management decision is likely to make working in that group least likely to suck over the next few months? If it were me, I would invest in technology and hires/transfers, not in bureaucracy. I would give that group everything it needs to build and rebuild that system twice. After all, doesn't Microsoft have the money for that?
- dtf 17y ago
  Actually I think the question on many people's lips is not whether whether some "mostly competent guy" will be scapegoated, but whether VP of Mobile Experiences Roz Ho will manage to keep her job. First we had the leaked details of the dire status of Microsoft's Pink project, and now this. Sure wouldn't like to be in her shoes right now.
jsz0 17y ago
While this is very unfortunate for SideKick users I suddenly feel pretty good about my own backup solution. Redundancy, rsync weekly to a live volume, multiple offline backups including one off-site (although it's a few months out of date right now. The important data exists on at least half a dozen sources at any given time. I am of course referring to my personal MP3 collection.
jws 17y ago
You see teacher, I have two dogs...
10ren 17y ago
This highlights a very reasonable fear that people and corporations have about the cloud. It might even kill off some of the smaller cloud-based players...
chanux 17y ago
Reminds me of something happend to a friend. Friend: Sir, my hard disk partition crashed and the project files are all gone. Lecturer: Don't you have a back up? Friend: The backup was on the same partition.

15 points by chuck_taylor 17y ago

I find it almost unbelievable that there were no cyclic backups.

> Will this prompt Microsoft to review its policy for data backups? "This was an extraordinary situation [...]. We greatly apologize [...]. We have a full team working in parallel with the restoration team to augment the redundancy and stability of this acquired service over the long-term." That's got to be the best non-answer answer I've read in a long time. And no - redundancy and stability of the service have nothing to do with the backup policy.

dlsspy 17y ago
You forgot "acquired." It was them. Those bastards we bought. I mean, sure that was almost a year ago, and most of the original employees are gone, but do keep in mind when we're talking about this that it's not a proper MS offering. Unless you like it. Then yay us.

brown9-2 17y ago

Wish this article had more details than just "the backups were impacted too". A few months ago Google had a large multi-hour App Engine outage. Afterwards (within 24 hours?) they posted a detailed technical explanation/postmortem online for anyone interested to read, with a play-by-play that even went minute-by-minute. Microsoft/Danger owe their customers more of a real explanation.

InclinedPlane 17y ago

Wrong, wrong, wrong. If you have a backup that CAN be impacted by a problem in the main server then you do not have a backup at all, you have, at best, redundancy. The very reason to have a legitimate backup is that it cannot be damaged by the same mechanisms that can damage your primary data. It should be inert, if possible, or at least to some extent immutable, and it should be offsite. RAID is not a backup. SAN is not a backup. Mirroring to a redundant storage array is also not a true backup. Backups are hard to get right, and expensive. But the cost of getting it wrong can be monumental.

andrewvc 17y ago
I couldn't agree more. Real backups mean that at some level you can't delete that data, no matter how much you want to, without driving out to some secure remote location and demagnetizing some tapes.
- furyg3 17y ago
  Real backups mean that at some level you can't delete that data It goes even farther than that: Real backups mean that at some level you can recover that data

makecheck 17y ago

In my experience, new "policies" put in place in these situations don't really do anything. They just inject bureaucrats where there were none before, and aggravate the few people remaining who might really help. And you can bet managers will start a witch hunt, firing some mostly-competent guy as a scapegoat (which may "feel good" at the time, but ironically will only slow down their recovery and set them up for their next failure). The smart thing would be for Microsoft to think of the employees: what management decision is likely to make working in that group least likely to suck over the next few months? If it were me, I would invest in technology and hires/transfers, not in bureaucracy. I would give that group everything it needs to build and rebuild that system twice. After all, doesn't Microsoft have the money for that?

dtf 17y ago
Actually I think the question on many people's lips is not whether whether some "mostly competent guy" will be scapegoated, but whether VP of Mobile Experiences Roz Ho will manage to keep her job. First we had the leaked details of the dire status of Microsoft's Pink project, and now this. Sure wouldn't like to be in her shoes right now.

jsz0 17y ago

While this is very unfortunate for SideKick users I suddenly feel pretty good about my own backup solution. Redundancy, rsync weekly to a live volume, multiple offline backups including one off-site (although it's a few months out of date right now. The important data exists on at least half a dozen sources at any given time. I am of course referring to my personal MP3 collection.

jws 17y ago

You see teacher, I have two dogs...

10ren 17y ago

This highlights a very reasonable fear that people and corporations have about the cloud. It might even kill off some of the smaller cloud-based players...

chanux 17y ago

Reminds me of something happend to a friend. Friend: Sir, my hard disk partition crashed and the project files are all gone. Lecturer: Don't you have a back up? Friend: The backup was on the same partition.