8 ms·

I’ve been a customer for years and recently had a catastrophic failure of a computer and it’s direct attached backup drive. I have spent the last four days wai

I’ve been a customer for years and recently had a catastrophic failure of a computer and it’s direct attached backup drive. I have spent the last four days waiting for backblaze to create a restore a backup for a computer on and last I checked it was at 9%. I chatted with support and they said this is normal. That progress may jump to completed any time because the restore only tracks file count. I wish I could say I’m happy about this but I’m four days into trying to pull down my data and I still have no idea when I’ll be able to start the download. This sucks.

monochromatic 9y ago
Why would it take any time at all to "create a restore"?
- brianwski 9y ago
  Brian from Backblaze here. All the data is encrypted and spread throughout our datacenters. So we fetch each file and decrypt it (with the password you provided when kicking off the restore). If this is less than 1 GByte it is instantaneous. But if it is a TByte of data, it takes a few hours to decrypt and add to the ZIP file.
  - monochromatic 9y ago
    That makes sense, thanks for the response. (4 days and counting still seems excessive though.)
    - brianwski 9y ago
      Definitely seems excessive. We'll look into it. There are some things that can affect it. We have a "pool" of restore servers that do this task. The oldest ones in the pool are 9 year old computers with slow hard drives. The newest restore servers are built on SSDs and are blazingly fast and have newer processors. So based on which restore server was assigned the task, plus other things like the size of each file can speed or slow the restore. We are in the middle of a project to speed up all restores. Hilariously we figured out that simply by decommissioning the oldest, saddest restore servers the restores averaged FASTER restore times.
      - BillinghamJ 9y ago
        Presumably you mean faster than the non-hardware solution you were working on?
  - explainplease 9y ago
    This is confusing and concerning to me as a potential customer. Having used CrashPlan and other proprietary and FOSS backup software, I don't understand this description of how restoration works with Backblaze. 1. Why is the data being decrypted server-side instead of client-side? The server should be sending the encrypted data to the client, and the password should never leave the client machine. 2. Why are zip files being created? The client software should be receiving the data and writing the files directly to the selected restore directory. e.g. this is how restore works with CrashPlan, and it only takes a few seconds to begin the process. Files are written directly to the selected output directory.
    - brianwski 9y ago
      I'll answer these in reverse order: > 2. Why are zip files being created? We actually offer two forms of restore: A) Zip File Download, and B) External USB Hard Drive FedEx'ed to your home. In the case of the Zip file download, we chose the format of zip because both Mac and PC (the most common desktops) natively understand it with no additional software needed. In other words, if you just lost your computer, go to ANY COMPUTER ANYWHERE and you can fetch your files with a web browser. Zip preserves the file hierarchy and the last modified time, so it's "pretty good". In the case of the USB Hard Drive there is no zip file, we can correctly place each file with all the correct timestamps in the correct heirarchy on the USB Hard Drive (which is actually an encrypted hard drive) and then FedEx the hard drive to you anywhere in the world in a day or two. > 1. Why is the data being decrypted server-side instead of client-side? Short Answer: Ease of use. Longer Answer: To clarify, Backblaze produces four different products/modes for different customers with different needs and requirements. We want customers to choose what is appropriate for them. One size does not fit all: 1) Online Backup ($5/month) where every file is encrypted on your laptop BEFORE being sent to Backblaze and your backup is secured by your username/password - where you can recover your password if you have access to your email account. (We support two-factor auth which provides an additional optional layer of protection.) 2) Online Backup ($5/month) where every file is encrypted on your laptop BEFORE being sent to Backblaze and your backup is secured by your username/password AND your private encryption key is secured by a "passphrase" that is not recoverable in any way, shape, or form. (Two-factor auth is also optional here.) 3) B2 Object storage (half of 1 cent/GByte/month) where you store your file completely unencrypted, and this can be "private" (only accessible by username/password) or "totally public accessible by knowing the URL". A good application of this is serving up a web page to the public - you really WANT people to see all the contents! 4) B2 Object storage (half of 1 cent/GByte/month) where Backblaze has zero knowledge. You cannot browse your file hierarchy because Backblaze doesn't know your filenames. You cannot preview your images. You cannot recover your passwords. There is no other option other than downloading the encrypted blobs and applying whatever decryption algorithm you decided on (we have no ability to know what that is). Ok, so I think some (many?) people in the security field think that Backblaze should ONLY offer mode #4 (and maybe #3 to serve up public websites). I happen to disagree and I personally feel that products #1 and #2 are useful and appropriate for some customers. But everybody is welcome to their opinion and we want to be completely open as to what exactly is occurring and what we are offering as a service. Personally I think #2 is an excellent trade off of security vs convenience. Your data is as impervious to attack as a zero knowledge system in #4 for years upon years. Then one day your laptop is stolen or crashes and you want your files back. You want all 4 TBytes back - so you order one of our free (encrypted) USB hard drives to be FedEx'ed to your home with all your data. To kick this process off FOR THE FIRST TIME EVER you tell us your passphrase (up until this very moment it really has been zero knowledge). At this moment you are opening a window of SLIGHTLY lowered security that slams shut after a few hours. For those few hours of preparing your 4 TByte restore, if an undetected hacker had compromised the one restore server in the Backblaze data center that your job was on, that hacker could possibly get access to your files. But then the reduced security window slams shut, we NEVER write your passphrase to any disk so it has now vaporized and we do not remember it, and if a hacker hacks into our system the following day you are STILL completely impervious. But again, as long as you fully understand the implications of #3 I am COMPLETELY supportive if you instead choose #4 which is our "Zero Knowledge" offering.
7b64f0f2 9y ago
Makes me wonder how much data/files you are trying to get back from their storage.
- finnh 9y ago
  Then why not ask?
rtpg 9y ago
I had a similar thing happen to me, what I did was pick the files I wanted immediately and got a zip file with those within... 30 minutes? Then got more of the files later
atYevP 9y ago
Yev from Backblaze here -> That's definitely not normal, how much data/files do you have (are you doing a .zip or a USB restore)? We're currently QAing a dramatic increase to the restores, but right now only available internally until we are confident in it. In the mean time though you can download mission-critical files in smaller-sized restored, that might help you get up and running faster while the mega-restore completes.
- godzillabrennus 9y ago
  It’s nearly 500GB of data. Email me Hn <at> strapr.com if you want more details
  - unixhero 9y ago
    Which is nothing
    - Spooky23 9y ago
      It's alot of data when the application is typically write-only, and throws little chunks of data all over the place!
- biomene 9y ago
  Do you reckon the increase in restores has anything to do with APFS? I also lost my computer and Time Machine this week after upgrading to High Sierra, had to restore from backblaze. Btw, unlike parent comment, I had a great experience with the restore and couldn't be happier with the decision to go with backblaze.
  - burnte 9y ago
    HE means they're QA testing a new method of restoring that will dramatically increase restore speed, not that there's a sudden uptick in restores. It's unclearly written, I had to read it twice before I got it.
BadassFractal 9y ago
That's the thing that always terrifies me about Backblaze. I really want to like it, the pricing is good and they are so open about how their business and technology operates, yet you constantly hear of people on HN getting bitten by corner cases where they lose their data :(
conception 9y ago
I <3 Backblaze, but for giant datasets their policies just don't match up with the reality of doing a full restore. I had ~11-!5TB with them and needed to pull it all down. With the speed of their website, inability to check what I'd restored already and time to get a restore via the mail and over the net, I definitely lost some data since it took longer than 30 days. For Grandma, they are fantastic. For big data sets, I can't trust them anymore. Note: They recently released a new website and software version that's supposed to address some of these issues but fool me once and all...
- dexterdog 9y ago
  Assuming you are paying $50/year to store well over 10TB of data I don't think you are the typical use case. I know they say that it's unlimited, but nothing ever is.
woolvalley 9y ago
This is why you always want the onsite & offsite backup, because offsite cloud backups will always be fairly slow to fully restore. I use a local NAS to achieve automatic onsite backups myself. It's part of the 3-2-1 backup rule.
- godzillabrennus 9y ago
  I had a Mac Pro Mid-2010 workstation with a RAID 5 Array configured to have a hot spare for redundancy. Then I had a locally attached hard drive for time machine backups + BackBlaze + a hosted repo for the work I do + I use a third party encrypted cloud file sharing service for the most important documents. The problem started when I decided to upgrade to High Sierra. The upgrade was catastrophic for this configuration of Mac Pro. I have been on the horn with an Apple Senior Advisor within Apple Enterprise Support who has escalated my specific issues to Apple Engineering. I decided to roll back but in the process of restoring the machine from the time machine backup the locally attached drive failed. Apple has yet to tell me if they will even support the Apple RAID controller even though this version of the Mac Pro is listed as compatible. In the meantime I need to get back up and running. This is why I pay for a cloud backup in addition to having a local backup. Fortunately, I also have a MacBook Pro laptop that I keep a duplicate copy of some of the data on. My productivity will not take a hit. It's just that most people are not me and it seems like it should not take more than four days to create a restore so that I can download my data.
  - e40 9y ago
    For the life of me, I can't figure out why people upgrade to new operating systems so quickly. I never upgrade my macOS until x.x.4, but I prefer x.x.5. I'm on 10.11.6 because 1) it works perfectly, and 2) Karabiner doesn't work on 10.12.
    - godzillabrennus 9y ago
      I routinely help others with operating system upgrades in my day to day so out of necessity I upgrade as soon as a general release becomes available. It's also why I have multiple layers of backups for my data.
godzillabrennus 9y ago
Brian from BackBlaze reached out and provided some feedback: I checked into it, and we think your restore is slow for several reasons: 1) Lots of tiny files (we are fixing this in a rewrite literally this week) 2) It landed on a slow restore server. (We are planning on decommissioning that server soon because it is 8 years old and slow.) We have the technology to move your restore to a faster restore server. However, the restore experts here tell me your restore is actually 90% through the part that would speed up, so they recommend just "let it run". I really really hope it will be finished and ready for you in the next few days. If you need some of your files any faster, I think you should prepare several smaller restores. It turns out that if you prepare four restores, each one 25% as large as the original, the four restores will complete 4x faster because they are prepared by four separate servers. You can create up to 10 simultaneous ZIP restores, and they are totally free, so feel free to spin up a few more restores even if they overlap with the original large 1.1 TByte restore. Please reach out to me again if your restore isn't ready by Tuesday (October 10th).
- d3ad1ysp0rk 9y ago
  Would it make sense for BackBlaze to offer a tool which automatically split restores into 10 zips if over a certain size?
deleted 9y ago
[deleted]

monochromatic 9y ago

Why would it take any time at all to "create a restore"?

brianwski 9y ago
Brian from Backblaze here. All the data is encrypted and spread throughout our datacenters. So we fetch each file and decrypt it (with the password you provided when kicking off the restore). If this is less than 1 GByte it is instantaneous. But if it is a TByte of data, it takes a few hours to decrypt and add to the ZIP file.
- monochromatic 9y ago
  That makes sense, thanks for the response. (4 days and counting still seems excessive though.)
  - brianwski 9y ago
    Definitely seems excessive. We'll look into it. There are some things that can affect it. We have a "pool" of restore servers that do this task. The oldest ones in the pool are 9 year old computers with slow hard drives. The newest restore servers are built on SSDs and are blazingly fast and have newer processors. So based on which restore server was assigned the task, plus other things like the size of each file can speed or slow the restore. We are in the middle of a project to speed up all restores. Hilariously we figured out that simply by decommissioning the oldest, saddest restore servers the restores averaged FASTER restore times.
    - BillinghamJ 9y ago
      Presumably you mean faster than the non-hardware solution you were working on?
- explainplease 9y ago
  This is confusing and concerning to me as a potential customer. Having used CrashPlan and other proprietary and FOSS backup software, I don't understand this description of how restoration works with Backblaze. 1. Why is the data being decrypted server-side instead of client-side? The server should be sending the encrypted data to the client, and the password should never leave the client machine. 2. Why are zip files being created? The client software should be receiving the data and writing the files directly to the selected restore directory. e.g. this is how restore works with CrashPlan, and it only takes a few seconds to begin the process. Files are written directly to the selected output directory.
  - brianwski 9y ago
    I'll answer these in reverse order: > 2. Why are zip files being created? We actually offer two forms of restore: A) Zip File Download, and B) External USB Hard Drive FedEx'ed to your home. In the case of the Zip file download, we chose the format of zip because both Mac and PC (the most common desktops) natively understand it with no additional software needed. In other words, if you just lost your computer, go to ANY COMPUTER ANYWHERE and you can fetch your files with a web browser. Zip preserves the file hierarchy and the last modified time, so it's "pretty good". In the case of the USB Hard Drive there is no zip file, we can correctly place each file with all the correct timestamps in the correct heirarchy on the USB Hard Drive (which is actually an encrypted hard drive) and then FedEx the hard drive to you anywhere in the world in a day or two. > 1. Why is the data being decrypted server-side instead of client-side? Short Answer: Ease of use. Longer Answer: To clarify, Backblaze produces four different products/modes for different customers with different needs and requirements. We want customers to choose what is appropriate for them. One size does not fit all: 1) Online Backup ($5/month) where every file is encrypted on your laptop BEFORE being sent to Backblaze and your backup is secured by your username/password - where you can recover your password if you have access to your email account. (We support two-factor auth which provides an additional optional layer of protection.) 2) Online Backup ($5/month) where every file is encrypted on your laptop BEFORE being sent to Backblaze and your backup is secured by your username/password AND your private encryption key is secured by a "passphrase" that is not recoverable in any way, shape, or form. (Two-factor auth is also optional here.) 3) B2 Object storage (half of 1 cent/GByte/month) where you store your file completely unencrypted, and this can be "private" (only accessible by username/password) or "totally public accessible by knowing the URL". A good application of this is serving up a web page to the public - you really WANT people to see all the contents! 4) B2 Object storage (half of 1 cent/GByte/month) where Backblaze has zero knowledge. You cannot browse your file hierarchy because Backblaze doesn't know your filenames. You cannot preview your images. You cannot recover your passwords. There is no other option other than downloading the encrypted blobs and applying whatever decryption algorithm you decided on (we have no ability to know what that is). Ok, so I think some (many?) people in the security field think that Backblaze should ONLY offer mode #4 (and maybe #3 to serve up public websites). I happen to disagree and I personally feel that products #1 and #2 are useful and appropriate for some customers. But everybody is welcome to their opinion and we want to be completely open as to what exactly is occurring and what we are offering as a service. Personally I think #2 is an excellent trade off of security vs convenience. Your data is as impervious to attack as a zero knowledge system in #4 for years upon years. Then one day your laptop is stolen or crashes and you want your files back. You want all 4 TBytes back - so you order one of our free (encrypted) USB hard drives to be FedEx'ed to your home with all your data. To kick this process off FOR THE FIRST TIME EVER you tell us your passphrase (up until this very moment it really has been zero knowledge). At this moment you are opening a window of SLIGHTLY lowered security that slams shut after a few hours. For those few hours of preparing your 4 TByte restore, if an undetected hacker had compromised the one restore server in the Backblaze data center that your job was on, that hacker could possibly get access to your files. But then the reduced security window slams shut, we NEVER write your passphrase to any disk so it has now vaporized and we do not remember it, and if a hacker hacks into our system the following day you are STILL completely impervious. But again, as long as you fully understand the implications of #3 I am COMPLETELY supportive if you instead choose #4 which is our "Zero Knowledge" offering.

7b64f0f2 9y ago

Makes me wonder how much data/files you are trying to get back from their storage.

finnh 9y ago
Then why not ask?

rtpg 9y ago

I had a similar thing happen to me, what I did was pick the files I wanted immediately and got a zip file with those within... 30 minutes? Then got more of the files later

atYevP 9y ago

Yev from Backblaze here -> That's definitely not normal, how much data/files do you have (are you doing a .zip or a USB restore)? We're currently QAing a dramatic increase to the restores, but right now only available internally until we are confident in it. In the mean time though you can download mission-critical files in smaller-sized restored, that might help you get up and running faster while the mega-restore completes.

godzillabrennus 9y ago
It’s nearly 500GB of data. Email me Hn <at> strapr.com if you want more details
- unixhero 9y ago
  Which is nothing
  - Spooky23 9y ago
    It's alot of data when the application is typically write-only, and throws little chunks of data all over the place!
biomene 9y ago
Do you reckon the increase in restores has anything to do with APFS? I also lost my computer and Time Machine this week after upgrading to High Sierra, had to restore from backblaze. Btw, unlike parent comment, I had a great experience with the restore and couldn't be happier with the decision to go with backblaze.
- burnte 9y ago
  HE means they're QA testing a new method of restoring that will dramatically increase restore speed, not that there's a sudden uptick in restores. It's unclearly written, I had to read it twice before I got it.

BadassFractal 9y ago

That's the thing that always terrifies me about Backblaze. I really want to like it, the pricing is good and they are so open about how their business and technology operates, yet you constantly hear of people on HN getting bitten by corner cases where they lose their data :(

conception 9y ago

I <3 Backblaze, but for giant datasets their policies just don't match up with the reality of doing a full restore. I had ~11-!5TB with them and needed to pull it all down. With the speed of their website, inability to check what I'd restored already and time to get a restore via the mail and over the net, I definitely lost some data since it took longer than 30 days. For Grandma, they are fantastic. For big data sets, I can't trust them anymore. Note: They recently released a new website and software version that's supposed to address some of these issues but fool me once and all...

dexterdog 9y ago
Assuming you are paying $50/year to store well over 10TB of data I don't think you are the typical use case. I know they say that it's unlimited, but nothing ever is.

woolvalley 9y ago

This is why you always want the onsite & offsite backup, because offsite cloud backups will always be fairly slow to fully restore. I use a local NAS to achieve automatic onsite backups myself. It's part of the 3-2-1 backup rule.

godzillabrennus 9y ago
I had a Mac Pro Mid-2010 workstation with a RAID 5 Array configured to have a hot spare for redundancy. Then I had a locally attached hard drive for time machine backups + BackBlaze + a hosted repo for the work I do + I use a third party encrypted cloud file sharing service for the most important documents. The problem started when I decided to upgrade to High Sierra. The upgrade was catastrophic for this configuration of Mac Pro. I have been on the horn with an Apple Senior Advisor within Apple Enterprise Support who has escalated my specific issues to Apple Engineering. I decided to roll back but in the process of restoring the machine from the time machine backup the locally attached drive failed. Apple has yet to tell me if they will even support the Apple RAID controller even though this version of the Mac Pro is listed as compatible. In the meantime I need to get back up and running. This is why I pay for a cloud backup in addition to having a local backup. Fortunately, I also have a MacBook Pro laptop that I keep a duplicate copy of some of the data on. My productivity will not take a hit. It's just that most people are not me and it seems like it should not take more than four days to create a restore so that I can download my data.
- e40 9y ago
  For the life of me, I can't figure out why people upgrade to new operating systems so quickly. I never upgrade my macOS until x.x.4, but I prefer x.x.5. I'm on 10.11.6 because 1) it works perfectly, and 2) Karabiner doesn't work on 10.12.
  - godzillabrennus 9y ago
    I routinely help others with operating system upgrades in my day to day so out of necessity I upgrade as soon as a general release becomes available. It's also why I have multiple layers of backups for my data.

godzillabrennus 9y ago

Brian from BackBlaze reached out and provided some feedback: I checked into it, and we think your restore is slow for several reasons: 1) Lots of tiny files (we are fixing this in a rewrite literally this week) 2) It landed on a slow restore server. (We are planning on decommissioning that server soon because it is 8 years old and slow.) We have the technology to move your restore to a faster restore server. However, the restore experts here tell me your restore is actually 90% through the part that would speed up, so they recommend just "let it run". I really really hope it will be finished and ready for you in the next few days. If you need some of your files any faster, I think you should prepare several smaller restores. It turns out that if you prepare four restores, each one 25% as large as the original, the four restores will complete 4x faster because they are prepared by four separate servers. You can create up to 10 simultaneous ZIP restores, and they are totally free, so feel free to spin up a few more restores even if they overlap with the original large 1.1 TByte restore. Please reach out to me again if your restore isn't ready by Tuesday (October 10th).

d3ad1ysp0rk 9y ago
Would it make sense for BackBlaze to offer a tool which automatically split restores into 10 zips if over a certain size?

deleted 9y ago

[deleted]