5 ms·
"It's the (Backblaze) restores that fail."
- SeanKilleen 13y agoOne other note: I think it's important to say that I'm a HUGE fan of what Backblaze does in terms of open-sourcing their storage box designs and how they accomplish things. I think that's a noble thing to do, and have always thought highly of them for that.
- SeanKilleen 13y agoI wanted to post this here to bring attention to what happens when trusting a cloud backup provider with your data, as well as to spur Backblaze to hopefully pay more attention to this part of their business model. Again, while I say it in the post, I think it's important to state again that the CEO has been great in allowing a dialog via Twitter. This is not a post meant to diss Backblaze as much as it is to share concerns and put the thought out there for consideration. As the CEO Mr. Budman notes in the Storify link, they no doubt do hundreds of successful restores every day. But when I have two working systems fail for Backblaze restores but nothing else, it's still a cause of concern for me personally.
- voltagex_ 13y agoI wonder whether it's a header or an index on the compressed file, or the data itself that's damaged. Still not something you want from your backups.
- SeanKilleen 13y agoCorrect. They appear to basically zip the data using 7-zip and then transfer it, but I'm not sure what sort of hashing/checking they do to make sure the file makes it to me successfully. Very possible that the next time I download the 6GB file it will work. But given how long it takes to do that, 1) I'd like to know it will work and 2) tougher for me to consider utilizing the free option of doing this for all ~400 GB of data.
- howellnick 13y agoComing from working in IT support, it's frustrating to me that you've made a big public ordeal over something that, firstly was fixed by rebooting your PC, and secondly you haven't tried downloading the file a second time. I'm not necessarily defending Backblaze here, but posting this when it doesn't work quite right the first time seems a bit harsh. I feel like we need more information before we all grab our pitchforks.
- SeanKilleen 13y agoI agree with your point to a certain extent -- I've spent many years in IT Support as well. With that said: * I went out of my way to avoid Pitchforks, merely calling attention to an issue and also ensuring I blamed myself enough * A 6GB download took 12+ hours. How many times am I supposed to wait for critical data that long to have it fail, and what happens when I try my 340 GB restoration, wait weeks, and it fails? * I think there are many potential better ways of doing it -- breaking into chunks and verifying each chunk as it is downloaded, any sort of check before the application says "complete", a little more finesse than merely using 7-Zip, etc. There are ways the experience could be improved is the point I'm making, since as customers this is the part where Backblaze really earns its keep. I have no desire for pitchforks, which is why I pointed out that the CEO was extremely helpful and that I'm hopeful the problem will be resolved. This is not an attempt to shame / dump on Backblaze; if they resolve the problem I'll very much still recommend and use them. However, both sides of the issue (my idiocy/negligence and Backblaze's areas for improvement) seem to be worth discussing.
- howellnick 13y agoI don't use Backblaze, so I don't know what kind of upload speeds they have on restores, but 12+ hours for 6gb seems very slow (or the bottleneck is your download speed). It doesn't seem all that unreasonable for someone that's storing 340gb to download 6gb twice. Of course, and it's been stated already, that if this is critical data you should really keep another backup. I see that you've made some effort to be transparent and fair, but I think this should have stayed as conversation between yourself and Backblaze until there's more substance to the issue at hand. With that said, it sounds like there could be some good discussion on their restoration process.
- goldenkey 13y agoPlease don't singly use cloud backups. Your first line should be a local backup solution like a Time Capsule or other NAS device. Blind trust in a black box backup is not prudent.
- SeanKilleen 13y agoPrecisely the point I'm making. It was hubris and laziness and I'm surprised that I let myself go that long. In a follow-up article I discuss an overview of what I'll be doing going forward: http://www.seankilleen.com/2014/01/my-newest-backup-system-process-lessons.html http://www.seankilleen.com/2014/01/my-newest-backup-system-p...
- Retric 13y agoThere is nothing wrong with trusting online backup providers provided your data is backed up on two or more of them and there not both using the same back-end (aka Amazon s3).
- SeanKilleen 13y agoI agree, and I would add to that: "as long as you also have a local source." That was my failure here. Had I backed up to a local source, the slow restore times / issues would be marginal at best. So there are two lessons from my perspective: Backblaze needs to get its restore scenarios fixed for folks like me, and I need to get my shit together as far as a back up strategy. Good lessons to be learned for all I think.
- sahaskatta 13y agoFor this very reason I've always backed-up to both an encrypted external HDD and another encrypted copy in the cloud.
- forgottenpass 13y agoWhenever a cloud service failure is posted to HN, there are always posts basically telling the OP that expecting the service you paid for is a silly idea. Why isn't the SaaS industry in general roundly criticized as the fly by night hacks they always turn out to be when someone gets burnt?
- Zhenya 13y agoWired or wireless connection? I would always do bit sensitive data over a wired connection. This goes without saying.
- SeanKilleen 13y agoWired connection during both attempts (even on the laptop). But you're right and it does go without saying.
- Dylan16807 13y agoNo it doesn't. Especially when it comes to HTTPS.
- Zhenya 13y ago"No it doesn't" what? That it's ok to trust https to keep a huge download completely error free?
- Dylan16807 13y agoIt does not go without saying that you should use wired connections instead of wireless. Wifi has a perfectly good error detection mechanism that gets layered on top of the TCP mechanism.
- mburns 13y agoOr over wireless. Absolutely. If there was data corruption in the HTTPS stream, it wouldn't decrypt correctly or the packet wouldn't checksum correctly and it would be retransmitted. TCP works just fine over wired vs wireless.
- mrchucklepants 13y agoThis is a little disconcerting. I have 3 computers backed up with Backblaze. I had a successful 350 GB restore last week. I hope that wasn't an anomaly.
- SeanKilleen 13y agoFrom what I gather, my situation may be the anomaly. Still concerning though. I'm still processing a 340 GB restore in the hopes that this one will go better. We'll see. Your success has me hopeful!
- bydo 13y agoYour situation is an anomaly, or this specific incident? You _have_ actually tested restoring from your only backup solution at a service you've been evangelizing for years, right?
- SeanKilleen 13y agoA fair point. Yes, I've tested and restored things before. However, those were normally smaller footprints and not the entirety of my data. Given the rest of the public's generally good experience with them that I could find (and kept seeing), I felt fairly confident that my data is safe (which is to say, I might not be able to get to it easily, but it's not going to disappear into thin air either). I'm thinking given prior experiences, other users, etc. that my experience currently is the anomaly (and hoping so).
- magic5227 13y agoFYI I left Backblaze for Crashplan for the same reasons. I believe Blackblaze also removes the backup after some number of hours, so if you don't download the backup by then, you're in big trouble. Crashplan also allows external and network drives to be backed up.
- SeanKilleen 13y agoCould you elaborate? Same reasons in that you had slow or broken restorations? Also curious: have you had to restore with CrashPlan, and if so, how has that process differed?
- magic5227 13y agoI had many issues with Backblaze over several years. I found 1-2 times a year, when initiating a backup, the entire archive I had disappeared, as in, my computer would start to send all of my files all over again. This was very disconcerting. I never liked seeing all files on an network drive disappear just because it was unplugged. The final straw was when I attempted to restore a large archive, and saw the download speed was so slow, it would not finish before this restore was removed on Backblaze's servers. Crashplan, has had zero issues over 2-3 years now. I've restored large files without issue, they store versions which I don't even need/want, and the desktop software works very well. Highly recommended.
- durandal1 13y agoTheir selling point is "Unlimited data backup for your Mac or PC, just $5/month." If you're in tech that should tell you it's not for storing "mission critical data" (paraphrasing the blogpost). For $5/month you can also not expect round the clock, seven days a week personal attention. Don't get me wrong, Backblaze seems like a kickass service for an off-site backup of my personal data.
- SeanKilleen 13y agoA good point: "mission critical" is an overstatement and I should phrase that differently. "Critical" to me means personal value (i.e. family photos), and not necessarily in terms of timeliness (i.e. business related). And to clarify, I don't expect personal attention. Contacting the CEO was because I thought there might be a legitimate problem with a server and support hadn't responded, and writing this blog came after I'd attempted to follow up with support but hadn't heard back for over 8 hours. I've stated this elsewhere on the thread but I'm by no means out to shame Backblaze. I think there are some UX points worth revisiting, and I may be looking at another provider, but I'm not trying to damn the company. I think it's a discussion worth having on how they might do things better (and also how I might do things better).
- hobs 13y agoI also lost a drive recently and found that while uploading to them was a breeze, the download was coming in at 14KB/s. I thought I was the one having a problem, but maybe not.
- SeanKilleen 13y agoYep, you weren't. Glad to know I wasn't the only one. Speeds varied for me, though the transfer did take much longer than the estimates. And that makes sense, given that I'm sure it's a ton of bandwidth to deal with all of those multiple restores. All the same, setting some expectations in those situations would be helpful, too.
- free652 13y agoMy current backup strategy: MacBook & Mini -> Time Machine on Mini (second hd), everything is using FileVault Pictures -> Bitcasa & local copies Personal Documents -> TrueCrypt Volume on Dropbox with local copies, coding projects are on dropbox also, obviously I have code in SVN/GIT repositories. I have 2 local copies, one in the safe deposit box and rotate them about every 3 months. This is pictures + personal files on a FileVault volume. Currently I don't backup my windows gaming pc aside from game saves on Bitcasa.
- kyle_t 13y agoI had the exact same situation three years ago with Carbonite. Had 80GB of data that I was never able to be restore. Their only means of amending the situation was offering a year for free. Never again. I had a local backup but it was unfortunately 3+ months out of date. Now I keep everything locally backed up with a NAS and a mix of dropbox/google drive.
- manishsharan 13y agoBacking up your data is not enough; you must also occasionally restore your data ( to a temp location )just to make sure your backup media or its governing database are not silently screwing up your backed up data.
- SeanKilleen 13y agoAgree 100%. In the future I plan to do this with both my local and cloud personal backups. (I would always do this with any business-related backup; I don't know why I my standards be lower for my personal data.)
- xbryanx 13y agoDoes anyone have restore experience with Tarsnap? Tarsnap seems like a great service, but I've never had to do a restore.
- sc68cal 13y agoI've pulled individual files from my tarsnap archives, and did a full restore of a server into a VirtualBox VM to verify that everything worked OK and can confirm that tarsnap works as advertised.
- SeanKilleen 13y agoA question, because I'm unfamiliar with the way HN works in this regard. My article was #2 on the front page for a brief spell. Now I look, and it's nowhere to be found. Is this by design? I don't need it to be on the top page and this isn't about the page views, I'm just not sure how it would disappear. Did a moderator somewhere frown upon it?
- SeanKilleen 13y agoHaha, nevermind, I'm an idiot. Just hadn't considered that the sheer amount of content would knock it off rather quickly (which makes sense, given that this is not nearly worth the focus that some other topics are.)
- thaumaturgy 13y agoThere doesn't seem to be any attempt to confirm that this is actually a problem on Backblaze's end -- it's just as likely to be a problem with Sean's internet connection. One of the problems with downloading large zip archives is that a few scrambled bits or dropped packets can scratch the whole archive. Backblaze offers the ability to get your data in the form of a flash drive or external drive (https://help.backblaze.com/entries/20224632-how-can-i-restore-my-files https://help.backblaze.com/entries/20224632-how-can-i-restor...). It's not free, but if your data is important, it's a pretty reasonable cost-of-hardware-and-handling price.
- SeanKilleen 13y agoHey there! Thanks for commenting. I think I commented on this aspect in a few other places on the thread, but while I glossed over the IT troubleshooting steps in my blog post, I've been in IT for 10+ years in the roles of support, sysadmin, and developer. I _of course_ did some troubleshooting. Two different PCs with internet connections that worked completely reliably both had different issues, and they were both in known stable states at the time. On each, I'd tested the internet connection, speed, etc. Nothing else was wrong with my internet connection at the time, at least in terms of things like loading web pages, downloads, and streaming HD video. In the first case, the speed seemed to drop to zero and then timeout. In the second, case the file took a long time to download and then came out corrupted anyway. I'm not saying these things don't happen; but I'm saying that, if it's a company's bread and butter, it would make more sense to put as many safeguards in place to make sure that after 12+ hours, a corrupted file doesn't happen. Also, I'm aware that a few dropped packets can be an issue. This is why I used Backblaze's advertised downloader application -- I got the impression that it would provide better stability, checking, etc. to ensure that the zip file did not get corrupted. I also mention the external HDDs in my post, but my point is that I pay them so that I can restore my files conveniently, not so that I can be upsold to a $189 drive. Normally, I would drop the cash, but after just spending several hundred dollars on a new motherboard, I figured I would use their download mechanism, which is offered. I think I'm allowed to have a reasonable expectation that it would work. With all that said, this is for sure not some "grab your pitchforks!" moment. I'm doing this partly to call attention to my own stupidity at putting my faith in one cloud backup provider, and also to highlight that Backblaze's process for restoring could likely be a little bit smoother, given that this is when users would like things to be absolutely foolproof. I remain a fan of Backblaze's CEO who was responsive when support wasn't, and of the company itself which has done great things like open-sourcing its hardware. I've since been in contact with support again and I will be sure to update my post once the matters are resolved.