8 ms·

It's always important to remember the difference between a syncing service and a backup service. A syncing service sometimes feels like a backup, because you ca

by jpadvo 14y ago

It's always important to remember the difference between a syncing service and a backup service. A syncing service sometimes feels like a backup, because you can use it to recover files if a local device is destroyed or lost. HOWEVER, any service capable of syncing files is equally capable of destroying them. It's important to have an automated one-way backup system that you can manually restore from. Something like Tarsnap [1] looks like a really good possibility (I haven't used it myself, but it seems solid) [1] http://www.tarsnap.com/gettingstarted.html http://www.tarsnap.com/gettingstarted.html

erichocean 14y ago
HOWEVER, any service capable of syncing files is equally capable of destroying them. That's simply not true (I've built sync systems that are incapable of destroying files). IMO, syncing designs that do have unrevokable overwrites are inherently brittle. (I don't know if Dropbox is built that way, but AFIAK, iCloud (and MobileMe before it) is -- and it sucks.)
- Dylan16807 14y ago
  Dropbox is revocable, the issue is for how long. In addition to online versions, the client holds on to old versions for three days. It used to be longer but the cache could get unreasonably large.
jonursenbach 14y ago
I've been a very happy Backblaze customer for over a year now. http://www.backblaze.com/ http://www.backblaze.com/
- jrs235 14y ago
  If backblaze backs up my dropbox folder and this dropbox bug affects my files that are getting backedup, if I don't catch the 0 byte files within the 4 week backblaze revision window, won't I still lose my files?
  - antidaily 14y ago
    Yep. And your wife will divorce you.
  - majormajor 14y ago
    Crashplan hangs on to deleted files for quite some time. I just used it last weekend to recover some stuff I'd deleted about a year ago and realized I wanted back. I was pleasantly surprised to find them still in the system. It has a lot of options for retaining old versions, too: http://support.crashplan.com/doku.php/reference/version_retention_and_backup_frequency http://support.crashplan.com/doku.php/reference/version_rete...
d0m 14y ago
To be fair, a good sync service with history is supposed to be a backup at the same time. I'm using git rather than dropbox but if I push something, it's there forever. So, I'd have to think that Dropbox being able to completely destroy files is a bug, not a feature. Maybe a more experienced dropbox user can correct me?
- bonzoesc 14y ago
  Why would a good sync service with history be a backup? They don't store a history indefinitely. Of course destroying files and history is a bug, but something that instantly replicates changes to those files from machine-to-machine shouldn't be considered a reliable way to recover them weeks or months later.
  - tlrobinson 14y ago
    Dropbox does have "Packrat unlimited undo history" for $4/month. I wonder if he could have recovered these files if he had this option.
  - Dylan16807 14y ago
    Does git push+pull not count as a "good sync service"?
- itcmcgrath 14y ago
  The difference is that a backup service shouldn't ever be able to corrupt your primary copy. Bugs happen. If a bug happens on the sync-ing service and it trashes your 'back-up'/history then syncs and trashes your primary copy, your toast. Sync'ing != Backup. They are for different problems and have different restrictions/pitfalls.
  - whichdan 14y ago
    The thing is, Dropbox treats their version as the primary copy. For instance, I added several directories to Dropbox, and decided that I didn't want it syncing a few of them. I de-selected the directories under "Selective Sync" and it removed them from my computer - even though my computer was the original source.
shuzchen 14y ago
"Permanently Lose Your Files" -> "I am able to restore each file from Dropbox's version history" It's also important to remember the meaning of the word "permanently", while this is a UX disaster, the title is misleading.
- throwit1979 14y ago
  ...except the author permanently lost files that were 0-byted before some sort of internal Dropbox process wiped the version history.
fusiongyro 14y ago
Tarsnap is amazing for Unix systems. Flat out, one of the coolest services I've ever used, and one of the cheapest. I'm backing up my VPS with it, and I can't recommend it enough. I have a friend that strongly recommends CrashPlan, but I haven't tried it out yet on my Mac. I'm curious to though.
- guygurari 14y ago
  I use CrashPlan both on Mac and on Windows. On the Mac it works very well. On Windows it is okay, though it needs some hand holding -- sometimes the service gets stuck and needs to be restarted manually. Overall I find that it is better than Mozy (a competing service).
  - mc32 14y ago
    If the service tends to hang, you might need to assign it more than the 512MB RAM it defaults to. The setting is controlled thru the CrashPlanService.ini file.
  - jwr 14y ago
    Seconded. I switched from Mozy and would not go back.
- rdl 14y ago
  I love Crashplan -- especially because with essentially the same client, you can do backups to a public service, a hosted business service (multiple machines, centrally managed keys, etc.) or an enterprise hosted-by-yourself service.
- bitcartel 14y ago
  Doesn't Tarsnap store your data on their S3 account, rather than your own? If so, if how do you get your data back from Amazon if Tarsnap vanished tomorrow?
  - fusiongyro 14y ago
    This isn't really a concern for me, but I'd probably email Colin and ask for it. The tarsnap program is open-source, presumably if he stopped hosting it the server-side component could be replaced. The client does all the work.
barik 14y ago
This distinction is really without a difference. Dropbox consistently advertises themselves as both a syncing service as well as a backup service. Their tag line is even "Dropbox - Secure backup, sync and sharing made easy," with secure backup being ahead of sync and sharing. [1] "Even if your computer has a meltdown, your stuff is always safe in Dropbox and can be restored in a snap. In fact, if you're using the Dropbox desktop application, your files are backed up several times. The primary copy on your computer's hard drive is synced online and that copy is then backed up again for safety (emphasis mine). If you are using Dropbox to sync files between multiple computers, your files are backed up on those computers as well. If that isn't enough, Dropbox also keeps backups of all of your deleted and changed files too. ... It's hard to imagine a scenario where Dropbox could lose your files. Hypothetically, let's say a nuclear bomb blows up the data centers where your files are saved. Even then, your files are still safe and sound on your computer and any other computers linked to your Dropbox account." Clearly, though, Dropbox appears to have lost files (at least if we take it on faith that "these are bugs in Dropbox's syncing logic"), despite the fact that I see no mushroom cloud nearby. [1] https://www.dropbox.com/help/122/en https://www.dropbox.com/help/122/en
- wlesieutre 14y ago
  >Even then, your files are still safe and sound on your computer and any other computers linked to your Dropbox account." Unless, as happened here, Dropbox erased the file and synced a blank version across all your computers. They they're safe and sound on any computer linked to your dropbox connection that hasn't been connected to the internet since the file got corrupted. If you catch it in the 30 day window when they keep old versions you're find, but there are files in my Dropbox that I don't use every 30 days.
  - barik 14y ago
    As you said, "Dropbox erased the file and synced a blank version across all your computers." I agree. As such, I'm not sure why individuals are so quick to blame the end user just yet. On the other hand, I don't want to immediately blame Dropbox just yet either. If you backup garbage (say, because you have disk corruption), then you can't blame Dropbox for backing up exactly what you told it to. And Dropbox does offer a premium Packrat service if you want file history indefinitely. Perhaps the user can be blamed for assuming that he/she would only need 30 days of history, but this is really contingent on who caused the corruption to happen in the first place -- and that's unknown at the moment. [1] [1] https://www.dropbox.com/help/113/en https://www.dropbox.com/help/113/en
sdizdar 14y ago
Yes - sync is not a backup but services as Dropbox are designed to be backup too (with their revision feature). So I think the underlaying problem here is that any backup/syncing system might have a bug (like this one) or there might be operator or user error (deleting your revision history is just a couple clicks away). Recovery oriented computing website has a lot good papers on this topic [1]. This is very similar to problems with outages on Amazon EC2 - yes Amazon cloud is great but in order to make your service highly available you do need to have standby system on some other cloud (for example, we run on Rackspace but our standbys are on Amazon). One approach to protect yourself against problems like this is to replicate/sync all your files from one cloud storage (your primary one) to some other cloud service (GDrive, SugarSync, Box, etc.). So should Dropbox have a bug, then you still have everything in other cloud service: including all revisions. Services like cloudHQ [2] (that is my baby) can replicate and sync all your files from Dropbox to, for example, GDrive. And of course cloudHQ has options like "two-way" sync, "don't replicate deletion", "backup" (weekly incremental are in folders - so your will be fine even if "revisions" feature fails), etc. [1] http://roc.cs.berkeley.edu/ http://roc.cs.berkeley.edu/ [2] http://cloudHQ.net http://cloudHQ.net
almost 14y ago
Tarsnap is really good, I use it from some of my servers to do backup. Maybe I could plug my new app here as well, tidy.io[1] lets you archive or backup your files directly to and from your Dropbox. Feedback is always appreciated! [1]: https://www.tidy.io/ https://www.tidy.io/
JohnsonB 14y ago
Tarsnap isn't the best choice in most cases, it's just by a HN user. But backblaze, carbonite, crashplan, spideroak are more practical/windows available options.
- notacylon 14y ago
  Tarsnap is great and secure. Not just any HN user, Colin Percival, security officer for Freebsd and a Putnam fellow. It's a very practical and well priced option.
  - JohnsonB 14y ago
    No, it's only for UNIX like OSs and is command line only, making it impractical or putting it out of reach of many users, even those on HN. Putting it as the drop in replacement for Dropbox for backup is simply inaccurate. The services I listed are more applicable. And Percival's stature is irrelevant here.
    - andrewflnr 14y ago
      You're the one who brought up that it's "just by a HN user", as if that was the most salient aspect of his life. And I don't think the top-level put Tarsnap forward as a drop-in replacement for Dropbox anyway.
rarrrrrr 14y ago
This is one of the reasons that SpiderOak (an encypted sync AND backup service) keeps historical versions of everything forever, until customers explicitly chose to remove them. For services that purge old versions and deleted files at 30 days, you lose if you don't notice a problem promptly. You can't be expected to be watchful over gigabytes of data; that's the whole point of a backup service.
- raphman 14y ago
  Just for the record (no big issue but I am accustomed to founders/employees disclosing their affiliation) "I'm Alan Fairless, a co-founder at SpiderOak" [1] [1] http://news.ycombinator.org/user?id=rarrrrrr http://news.ycombinator.org/user?id=rarrrrrr
- robryan 14y ago
  Dropbox give you the choice, you can pay and have the versions stored forever or accept the lower level of service with 30 days for free. I am sure you have calculated the economics on keeping the extra copied for all non paying customers, it is possible it would be less sustainable. As an aside, it seems Dropbox is biding on the [SpiderOak] keyword on Adwords, you should probably at least outbid them on your brand terms to reduce confusion/ misdirection for potential customers.
  - pja 14y ago
    As an aside, it seems Dropbox is biding on the [SpiderOak] keyword on Adword So they are. Is it legally acceptable to bid on competitor trademarks? I thought that was regarded as being over the line these days - anyone know for sure? If you look at the adwords link, it's clear that Dropbox is bidding on "competitor keywords" as a class.
    - robryan 14y ago
      Depending on the country there are different rules about this[1]. There are different rules for trademarks in Ad copy and in the keyword. In terms of a trademark in your keyword, this is only unacceptable in: Australia, Brazil, China, Hong Kong, Macau, New Zealand, North Korea, South Korea, or Taiwan, and only after the trademark holder files a complaint. [1] http://support.google.com/adwordspolicy/bin/answer.py?hl=en&answer=6118 http://support.google.com/adwordspolicy/bin/answer.py?hl=en&...
aaronpk 14y ago
Agreed. Early in my Dropbox days I lost some files due to my account going over the 2GB limit. Since then, I've treated Dropbox as a syncing server, not a backup service. My current scheme is to make rolling snapshots of my Dropbox folder backed up to a local RAID array which backs up to a separate RAID array nightly. More info: http://aaronparecki.com/2010/190/article/1/how-to-back-up-dropbox-automatically-daily-weekly-monthly-snapshots http://aaronparecki.com/2010/190/article/1/how-to-back-up-dr... Hackernews link for that post if you're in to that sort of thing: http://news.ycombinator.com/item?id=4704667 http://news.ycombinator.com/item?id=4704667
- wycx 14y ago
  Good idea. You could also add a bit to the script explicitly looking for zero byte files and alert yourself to their formation.
hnriot 14y ago
There really is no difference between syncing and backup. Think of the backup/restore cycle as a sync and it will be clear. Backup is one way sync, restore completes the bidirectionality of sync. Sync doesn't necessarily have the capability to destroy files, rsync has a switch to delete files that are locally deleted. However Dropbox is supposed to be rsync + rcs so this kind of problem is supposedly easy to fix by simply reverting to a good previous version.

by jpadvo 14y ago

erichocean 14y ago

HOWEVER, any service capable of syncing files is equally capable of destroying them. That's simply not true (I've built sync systems that are incapable of destroying files). IMO, syncing designs that do have unrevokable overwrites are inherently brittle. (I don't know if Dropbox is built that way, but AFIAK, iCloud (and MobileMe before it) is -- and it sucks.)

Dylan16807 14y ago
Dropbox is revocable, the issue is for how long. In addition to online versions, the client holds on to old versions for three days. It used to be longer but the cache could get unreasonably large.

jonursenbach 14y ago

I've been a very happy Backblaze customer for over a year now. http://www.backblaze.com/ http://www.backblaze.com/

jrs235 14y ago
If backblaze backs up my dropbox folder and this dropbox bug affects my files that are getting backedup, if I don't catch the 0 byte files within the 4 week backblaze revision window, won't I still lose my files?
- antidaily 14y ago
  Yep. And your wife will divorce you.
- majormajor 14y ago
  Crashplan hangs on to deleted files for quite some time. I just used it last weekend to recover some stuff I'd deleted about a year ago and realized I wanted back. I was pleasantly surprised to find them still in the system. It has a lot of options for retaining old versions, too: http://support.crashplan.com/doku.php/reference/version_retention_and_backup_frequency http://support.crashplan.com/doku.php/reference/version_rete...

d0m 14y ago

To be fair, a good sync service with history is supposed to be a backup at the same time. I'm using git rather than dropbox but if I push something, it's there forever. So, I'd have to think that Dropbox being able to completely destroy files is a bug, not a feature. Maybe a more experienced dropbox user can correct me?

bonzoesc 14y ago
Why would a good sync service with history be a backup? They don't store a history indefinitely. Of course destroying files and history is a bug, but something that instantly replicates changes to those files from machine-to-machine shouldn't be considered a reliable way to recover them weeks or months later.
- tlrobinson 14y ago
  Dropbox does have "Packrat unlimited undo history" for $4/month. I wonder if he could have recovered these files if he had this option.
- Dylan16807 14y ago
  Does git push+pull not count as a "good sync service"?
itcmcgrath 14y ago
The difference is that a backup service shouldn't ever be able to corrupt your primary copy. Bugs happen. If a bug happens on the sync-ing service and it trashes your 'back-up'/history then syncs and trashes your primary copy, your toast. Sync'ing != Backup. They are for different problems and have different restrictions/pitfalls.
- whichdan 14y ago
  The thing is, Dropbox treats their version as the primary copy. For instance, I added several directories to Dropbox, and decided that I didn't want it syncing a few of them. I de-selected the directories under "Selective Sync" and it removed them from my computer - even though my computer was the original source.

shuzchen 14y ago

"Permanently Lose Your Files" -> "I am able to restore each file from Dropbox's version history" It's also important to remember the meaning of the word "permanently", while this is a UX disaster, the title is misleading.

throwit1979 14y ago
...except the author permanently lost files that were 0-byted before some sort of internal Dropbox process wiped the version history.

fusiongyro 14y ago

Tarsnap is amazing for Unix systems. Flat out, one of the coolest services I've ever used, and one of the cheapest. I'm backing up my VPS with it, and I can't recommend it enough. I have a friend that strongly recommends CrashPlan, but I haven't tried it out yet on my Mac. I'm curious to though.

guygurari 14y ago
I use CrashPlan both on Mac and on Windows. On the Mac it works very well. On Windows it is okay, though it needs some hand holding -- sometimes the service gets stuck and needs to be restarted manually. Overall I find that it is better than Mozy (a competing service).
- mc32 14y ago
  If the service tends to hang, you might need to assign it more than the 512MB RAM it defaults to. The setting is controlled thru the CrashPlanService.ini file.
- jwr 14y ago
  Seconded. I switched from Mozy and would not go back.
rdl 14y ago
I love Crashplan -- especially because with essentially the same client, you can do backups to a public service, a hosted business service (multiple machines, centrally managed keys, etc.) or an enterprise hosted-by-yourself service.
bitcartel 14y ago
Doesn't Tarsnap store your data on their S3 account, rather than your own? If so, if how do you get your data back from Amazon if Tarsnap vanished tomorrow?
- fusiongyro 14y ago
  This isn't really a concern for me, but I'd probably email Colin and ask for it. The tarsnap program is open-source, presumably if he stopped hosting it the server-side component could be replaced. The client does all the work.

barik 14y ago

This distinction is really without a difference. Dropbox consistently advertises themselves as both a syncing service as well as a backup service. Their tag line is even "Dropbox - Secure backup, sync and sharing made easy," with secure backup being ahead of sync and sharing. [1] "Even if your computer has a meltdown, your stuff is always safe in Dropbox and can be restored in a snap. In fact, if you're using the Dropbox desktop application, your files are backed up several times. The primary copy on your computer's hard drive is synced online and that copy is then backed up again for safety (emphasis mine). If you are using Dropbox to sync files between multiple computers, your files are backed up on those computers as well. If that isn't enough, Dropbox also keeps backups of all of your deleted and changed files too. ... It's hard to imagine a scenario where Dropbox could lose your files. Hypothetically, let's say a nuclear bomb blows up the data centers where your files are saved. Even then, your files are still safe and sound on your computer and any other computers linked to your Dropbox account." Clearly, though, Dropbox appears to have lost files (at least if we take it on faith that "these are bugs in Dropbox's syncing logic"), despite the fact that I see no mushroom cloud nearby. [1] https://www.dropbox.com/help/122/en https://www.dropbox.com/help/122/en

wlesieutre 14y ago
>Even then, your files are still safe and sound on your computer and any other computers linked to your Dropbox account." Unless, as happened here, Dropbox erased the file and synced a blank version across all your computers. They they're safe and sound on any computer linked to your dropbox connection that hasn't been connected to the internet since the file got corrupted. If you catch it in the 30 day window when they keep old versions you're find, but there are files in my Dropbox that I don't use every 30 days.
- barik 14y ago
  As you said, "Dropbox erased the file and synced a blank version across all your computers." I agree. As such, I'm not sure why individuals are so quick to blame the end user just yet. On the other hand, I don't want to immediately blame Dropbox just yet either. If you backup garbage (say, because you have disk corruption), then you can't blame Dropbox for backing up exactly what you told it to. And Dropbox does offer a premium Packrat service if you want file history indefinitely. Perhaps the user can be blamed for assuming that he/she would only need 30 days of history, but this is really contingent on who caused the corruption to happen in the first place -- and that's unknown at the moment. [1] [1] https://www.dropbox.com/help/113/en https://www.dropbox.com/help/113/en

sdizdar 14y ago

Yes - sync is not a backup but services as Dropbox are designed to be backup too (with their revision feature). So I think the underlaying problem here is that any backup/syncing system might have a bug (like this one) or there might be operator or user error (deleting your revision history is just a couple clicks away). Recovery oriented computing website has a lot good papers on this topic [1]. This is very similar to problems with outages on Amazon EC2 - yes Amazon cloud is great but in order to make your service highly available you do need to have standby system on some other cloud (for example, we run on Rackspace but our standbys are on Amazon). One approach to protect yourself against problems like this is to replicate/sync all your files from one cloud storage (your primary one) to some other cloud service (GDrive, SugarSync, Box, etc.). So should Dropbox have a bug, then you still have everything in other cloud service: including all revisions. Services like cloudHQ [2] (that is my baby) can replicate and sync all your files from Dropbox to, for example, GDrive. And of course cloudHQ has options like "two-way" sync, "don't replicate deletion", "backup" (weekly incremental are in folders - so your will be fine even if "revisions" feature fails), etc. [1] http://roc.cs.berkeley.edu/ http://roc.cs.berkeley.edu/ [2] http://cloudHQ.net http://cloudHQ.net

almost 14y ago

Tarsnap is really good, I use it from some of my servers to do backup. Maybe I could plug my new app here as well, tidy.io[1] lets you archive or backup your files directly to and from your Dropbox. Feedback is always appreciated! [1]: https://www.tidy.io/ https://www.tidy.io/

JohnsonB 14y ago

Tarsnap isn't the best choice in most cases, it's just by a HN user. But backblaze, carbonite, crashplan, spideroak are more practical/windows available options.

notacylon 14y ago
Tarsnap is great and secure. Not just any HN user, Colin Percival, security officer for Freebsd and a Putnam fellow. It's a very practical and well priced option.
- JohnsonB 14y ago
  No, it's only for UNIX like OSs and is command line only, making it impractical or putting it out of reach of many users, even those on HN. Putting it as the drop in replacement for Dropbox for backup is simply inaccurate. The services I listed are more applicable. And Percival's stature is irrelevant here.
  - andrewflnr 14y ago
    You're the one who brought up that it's "just by a HN user", as if that was the most salient aspect of his life. And I don't think the top-level put Tarsnap forward as a drop-in replacement for Dropbox anyway.

rarrrrrr 14y ago

This is one of the reasons that SpiderOak (an encypted sync AND backup service) keeps historical versions of everything forever, until customers explicitly chose to remove them. For services that purge old versions and deleted files at 30 days, you lose if you don't notice a problem promptly. You can't be expected to be watchful over gigabytes of data; that's the whole point of a backup service.

raphman 14y ago
Just for the record (no big issue but I am accustomed to founders/employees disclosing their affiliation) "I'm Alan Fairless, a co-founder at SpiderOak" [1] [1] http://news.ycombinator.org/user?id=rarrrrrr http://news.ycombinator.org/user?id=rarrrrrr
robryan 14y ago
Dropbox give you the choice, you can pay and have the versions stored forever or accept the lower level of service with 30 days for free. I am sure you have calculated the economics on keeping the extra copied for all non paying customers, it is possible it would be less sustainable. As an aside, it seems Dropbox is biding on the [SpiderOak] keyword on Adwords, you should probably at least outbid them on your brand terms to reduce confusion/ misdirection for potential customers.
- pja 14y ago
  As an aside, it seems Dropbox is biding on the [SpiderOak] keyword on Adword So they are. Is it legally acceptable to bid on competitor trademarks? I thought that was regarded as being over the line these days - anyone know for sure? If you look at the adwords link, it's clear that Dropbox is bidding on "competitor keywords" as a class.
  - robryan 14y ago
    Depending on the country there are different rules about this[1]. There are different rules for trademarks in Ad copy and in the keyword. In terms of a trademark in your keyword, this is only unacceptable in: Australia, Brazil, China, Hong Kong, Macau, New Zealand, North Korea, South Korea, or Taiwan, and only after the trademark holder files a complaint. [1] http://support.google.com/adwordspolicy/bin/answer.py?hl=en&answer=6118 http://support.google.com/adwordspolicy/bin/answer.py?hl=en&...

aaronpk 14y ago

Agreed. Early in my Dropbox days I lost some files due to my account going over the 2GB limit. Since then, I've treated Dropbox as a syncing server, not a backup service. My current scheme is to make rolling snapshots of my Dropbox folder backed up to a local RAID array which backs up to a separate RAID array nightly. More info: http://aaronparecki.com/2010/190/article/1/how-to-back-up-dropbox-automatically-daily-weekly-monthly-snapshots http://aaronparecki.com/2010/190/article/1/how-to-back-up-dr... Hackernews link for that post if you're in to that sort of thing: http://news.ycombinator.com/item?id=4704667 http://news.ycombinator.com/item?id=4704667

wycx 14y ago
Good idea. You could also add a bit to the script explicitly looking for zero byte files and alert yourself to their formation.

hnriot 14y ago

There really is no difference between syncing and backup. Think of the backup/restore cycle as a sync and it will be clear. Backup is one way sync, restore completes the bidirectionality of sync. Sync doesn't necessarily have the capability to destroy files, rsync has a switch to delete files that are locally deleted. However Dropbox is supposed to be rsync + rcs so this kind of problem is supposedly easy to fix by simply reverting to a good previous version.