4 ms·

Hey Tedivm! Yev from Backblaze here -> It's true that the data was removed from your account. After your trial lapsed and was not paid for, we removed the data

by atYevP 13y ago

Hey Tedivm! Yev from Backblaze here -> It's true that the data was removed from your account. After your trial lapsed and was not paid for, we removed the data after 7 days as is our standard practice. Unfortunately in your case the overview page did not update to show that you had no data associated with your computer, so when you purchased the license, there was no data on the machine that it attached too, so when you erased data from the machine while performing maintenance, you assumed the data was on Backblaze, when it was in fact purged from our system a few weeks prior. We've since fixed that visual glitch (if you have a trial that expires and is not paid for, once the data has been deleted, that will be reflected on the overview page) AND we're currently working on some other notification systems to help people avoid losing data. As always, I apologize that you had this problem, but I'm happy to inform you that we've since fixed that visual bug on our end.

tedivm 13y ago
So everyone else is clear, we did pay for the account after the trial ended. So we were paying customers, this wasn't an incident where a trial expired and we tried to get our data back. Also, to be clear, the customer service experience was horrible. The support team repeatedly ignored what we were saying and send us form responses back which were just not relevant. The twitter team claimed we misread the retention policy. The whole thing was kind of ridiculous. Even more to the point, and the main reason why I tell this story, is that people should know how this company operates. People should know that they don't have backups of backups, that there is no margin for error should your account lapse, and that their systems in general are not robustly programmed in the sense that there are ways that your data could be lost and you wouldn't know it. From my perspective Backblaze only has one thing going for it over competing companies, and that's it's marketing department.
- novaleaf 13y ago
  well, at least you should be comforted by the fact that your problem (or at least, re-reporting it here on HN) will cause them to fix the bug that got you into this mess.
- graeme 13y ago
  Can anyone else comment on their experience with Backblaze's systems, and how competitors compare?
- TwoBit 13y ago
  If I understand it right, you purchased a license after your trial expired and the backup was purged. And you didn't backup anything new upon getting the new license. So the GUI status bug was a real and unfortunate bug on Backblaze's side, but it was only a GUI bug and in fact they didn't lose your data in any unexpected way. This situation seems to be about 2/3rds BackBlaze's fault and 1/3rd your fault. You also have bonus fault on your side for not testing the BackBlaze backup. Never ever trust a backup, even if it's right in front of you on your own disk, unless you can successfully restore it.
  - atYevP 13y ago
    We do recommend having a 3-2-1 backup (3 total copies, 2 local (1 original, 1 local external), and 1 offsite (where Backblaze comes in). In Tedivm's defense though, he did find an actual glitch in our UI that occurred in edge cases, and we've since fixed it. The unfortunate part is that he lost data when we no longer had a copy, and we're actively working on additional ways to avoid that in the future.
  - tedivm 13y ago
    Yup, there were serious flaws in our backup plans here. In my defense, we hadn't been dating long enough at the time for me to really get things set and we were on our way to burning man ;-) I am not claiming to be without fault, but I do think that Backblaze is on the lower end of the reliability spectrum when it comes companies that do this, which is why I bring up these issues. I've switched over to Crashplan and have external harddrives doing local backups. I should also point out that this isn't the only issue that's come up with this company. Apparently their client side GUI is also not completely truthful about when it's backed things up, and will claim to have backed up files when it's really only backed up the indexes. It then spreads the real backups out in order to increase performance. Although this is a cool feature, it's one people should be aware of and not hidden. I have less knowledge of this though, as I only saw it in a few blog posts (google around instead of taking my word for it).
    - ghshephard 13y ago
      "Lower end of spectrum" - of precisely two companies that I know of, CrashPlan and BackBlaze, that do "Whole Disk Backups at a fixed rate". I'm a big fan of using a combination of Arq (Love It), Dropbox, BackBlaze, and SuperDuper, but - crashplan/backblaze are both pretty similar, and both have their pros/cons, and, likewise, hundreds of anecdotes from the user community of each. For what it's worth, I'm in my fifth year of Backblaze, and I've always been able to recover all my files - but I've also kept my subscription paid up. And, I don't believe I've ever seen your "real backups happening spread over" claim - I am frequently in the field, so I am very aware of exactly when backblaze is sending data (frequently over my iPhone using Roaming Data) - and when backblaze says it's done backing up - that's pretty much exactly when data is no longer sent out.
icelancer 13y ago
How did you compensate tedivm for his issues, and how do you plan on fixing the support-related issues tedivm brought up, like your CSRs lying about the problems?
- atYevP 13y ago
  Icelancer, typically when users have any issues we offer a refund if they wish to leave the service, I can say that one was offered in this case as well. I do take issue with our "CSRs lying" though. Reading through the tickets, it is absolutely accurate that our dashboard was misrepresenting the data. It was showing Tedivm's data as still being there (or available), when in reality it was deleted weeks before the license was purchased for the account (after a trial expires we maintain data for 7 days, then remove it, though we do send alert emails stating that the trial is ending and the data will be removed). In Tedivm's case the license was purchased over 20 days after the trial expired. Our CSR didn't understand the issue that was occurring, because he had not experienced this particular problem before. The CSR was not deliberately trying to mislead anyone, and in truth once the issue was escalated and we realized was going on, we learned that Tedivm had a legitimate complaint, that we had a visual bug that was showing the data as still being there when it was not. Once we realized what was going on, we acknowledged the issue and we've since fixed that glitch as well, so now expired trials with their data deleted show that they have 0 files available online. I can go in to further into detail, but we try to keep all customer information/support queries private. We did do a bit more training with the CSRs about identifying atypical issues like this one, so hopefully they can be found out earlier in the support process instead of after a bit of back and forth as was the case this time.
  - brianwski 13y ago
    I think Yev put it well, but I just want to add that customer support has two distinct jobs (Tedivm's issue fell under #2): 1) Answering basic general questions like "how much does it cost"? 2) Debugging live customer issues. For example, if crappy anti-virus quarantines one of the Backblaze executables, what the heck is happening and why? Of if a customer's client cannot contact Backblaze's datacenter, which firewall is blocking it (maybe it is the software firewall on the computer, or maybe it is the router not allowing HTTPS through)? Etc.... In the process of debugging problems, the customer service reps are dealing with a truly insane array of possible issues and symptoms and they go back and forth with the customer asking for log files, ask if specific symptoms are happening, and in general COLLABORATE with the customer to get to the true bottom of the issue. Nobody lied. These are good, honest people that are really looking out for our customers and take their responsibilities seriously.

by atYevP 13y ago

tedivm 13y ago

So everyone else is clear, we did pay for the account after the trial ended. So we were paying customers, this wasn't an incident where a trial expired and we tried to get our data back. Also, to be clear, the customer service experience was horrible. The support team repeatedly ignored what we were saying and send us form responses back which were just not relevant. The twitter team claimed we misread the retention policy. The whole thing was kind of ridiculous. Even more to the point, and the main reason why I tell this story, is that people should know how this company operates. People should know that they don't have backups of backups, that there is no margin for error should your account lapse, and that their systems in general are not robustly programmed in the sense that there are ways that your data could be lost and you wouldn't know it. From my perspective Backblaze only has one thing going for it over competing companies, and that's it's marketing department.

novaleaf 13y ago
well, at least you should be comforted by the fact that your problem (or at least, re-reporting it here on HN) will cause them to fix the bug that got you into this mess.
graeme 13y ago
Can anyone else comment on their experience with Backblaze's systems, and how competitors compare?
TwoBit 13y ago
If I understand it right, you purchased a license after your trial expired and the backup was purged. And you didn't backup anything new upon getting the new license. So the GUI status bug was a real and unfortunate bug on Backblaze's side, but it was only a GUI bug and in fact they didn't lose your data in any unexpected way. This situation seems to be about 2/3rds BackBlaze's fault and 1/3rd your fault. You also have bonus fault on your side for not testing the BackBlaze backup. Never ever trust a backup, even if it's right in front of you on your own disk, unless you can successfully restore it.
- atYevP 13y ago
  We do recommend having a 3-2-1 backup (3 total copies, 2 local (1 original, 1 local external), and 1 offsite (where Backblaze comes in). In Tedivm's defense though, he did find an actual glitch in our UI that occurred in edge cases, and we've since fixed it. The unfortunate part is that he lost data when we no longer had a copy, and we're actively working on additional ways to avoid that in the future.
- tedivm 13y ago
  Yup, there were serious flaws in our backup plans here. In my defense, we hadn't been dating long enough at the time for me to really get things set and we were on our way to burning man ;-) I am not claiming to be without fault, but I do think that Backblaze is on the lower end of the reliability spectrum when it comes companies that do this, which is why I bring up these issues. I've switched over to Crashplan and have external harddrives doing local backups. I should also point out that this isn't the only issue that's come up with this company. Apparently their client side GUI is also not completely truthful about when it's backed things up, and will claim to have backed up files when it's really only backed up the indexes. It then spreads the real backups out in order to increase performance. Although this is a cool feature, it's one people should be aware of and not hidden. I have less knowledge of this though, as I only saw it in a few blog posts (google around instead of taking my word for it).
  - ghshephard 13y ago
    "Lower end of spectrum" - of precisely two companies that I know of, CrashPlan and BackBlaze, that do "Whole Disk Backups at a fixed rate". I'm a big fan of using a combination of Arq (Love It), Dropbox, BackBlaze, and SuperDuper, but - crashplan/backblaze are both pretty similar, and both have their pros/cons, and, likewise, hundreds of anecdotes from the user community of each. For what it's worth, I'm in my fifth year of Backblaze, and I've always been able to recover all my files - but I've also kept my subscription paid up. And, I don't believe I've ever seen your "real backups happening spread over" claim - I am frequently in the field, so I am very aware of exactly when backblaze is sending data (frequently over my iPhone using Roaming Data) - and when backblaze says it's done backing up - that's pretty much exactly when data is no longer sent out.

icelancer 13y ago

How did you compensate tedivm for his issues, and how do you plan on fixing the support-related issues tedivm brought up, like your CSRs lying about the problems?

atYevP 13y ago
Icelancer, typically when users have any issues we offer a refund if they wish to leave the service, I can say that one was offered in this case as well. I do take issue with our "CSRs lying" though. Reading through the tickets, it is absolutely accurate that our dashboard was misrepresenting the data. It was showing Tedivm's data as still being there (or available), when in reality it was deleted weeks before the license was purchased for the account (after a trial expires we maintain data for 7 days, then remove it, though we do send alert emails stating that the trial is ending and the data will be removed). In Tedivm's case the license was purchased over 20 days after the trial expired. Our CSR didn't understand the issue that was occurring, because he had not experienced this particular problem before. The CSR was not deliberately trying to mislead anyone, and in truth once the issue was escalated and we realized was going on, we learned that Tedivm had a legitimate complaint, that we had a visual bug that was showing the data as still being there when it was not. Once we realized what was going on, we acknowledged the issue and we've since fixed that glitch as well, so now expired trials with their data deleted show that they have 0 files available online. I can go in to further into detail, but we try to keep all customer information/support queries private. We did do a bit more training with the CSRs about identifying atypical issues like this one, so hopefully they can be found out earlier in the support process instead of after a bit of back and forth as was the case this time.
- brianwski 13y ago
  I think Yev put it well, but I just want to add that customer support has two distinct jobs (Tedivm's issue fell under #2): 1) Answering basic general questions like "how much does it cost"? 2) Debugging live customer issues. For example, if crappy anti-virus quarantines one of the Backblaze executables, what the heck is happening and why? Of if a customer's client cannot contact Backblaze's datacenter, which firewall is blocking it (maybe it is the software firewall on the computer, or maybe it is the router not allowing HTTPS through)? Etc.... In the process of debugging problems, the customer service reps are dealing with a truly insane array of possible issues and symptoms and they go back and forth with the customer asking for log files, ask if specific symptoms are happening, and in general COLLABORATE with the customer to get to the true bottom of the issue. Nobody lied. These are good, honest people that are really looking out for our customers and take their responsibilities seriously.