9 ms·

Dropbox Bug Can Permanently Lose Your Files

193 points by joshuacc 14y ago

c16 14y ago
Yep, I'm affected by this bug too.. Thankfully nothing important, but makes me wonder if I should leave my truecrypt folder on DropBox.
- gte910h 14y ago
  You should both distribute your file you want to keep to several devices, as well as a service who's job it is to back things up. Dropbox is only the former.
  - c16 14y ago
    I have a backup on a USB, but updating that daily is a hassle hence why I used DropBox- their added 2 step auth was also a big bonus, but if I lose that file I'll have a hard time getting my passwords back.. As jpadvo said, I might look into making some one way backups to S3 or something.
    - gte910h 14y ago
      Buy a time capsule and back it up locally or get a subscription to something like crash plan pro.
jpadvo 14y ago
It's always important to remember the difference between a syncing service and a backup service. A syncing service sometimes feels like a backup, because you can use it to recover files if a local device is destroyed or lost. HOWEVER, any service capable of syncing files is equally capable of destroying them. It's important to have an automated one-way backup system that you can manually restore from. Something like Tarsnap [1] looks like a really good possibility (I haven't used it myself, but it seems solid) [1] http://www.tarsnap.com/gettingstarted.html http://www.tarsnap.com/gettingstarted.html
- erichocean 14y ago
  HOWEVER, any service capable of syncing files is equally capable of destroying them. That's simply not true (I've built sync systems that are incapable of destroying files). IMO, syncing designs that do have unrevokable overwrites are inherently brittle. (I don't know if Dropbox is built that way, but AFIAK, iCloud (and MobileMe before it) is -- and it sucks.)
  - Dylan16807 14y ago
    Dropbox is revocable, the issue is for how long. In addition to online versions, the client holds on to old versions for three days. It used to be longer but the cache could get unreasonably large.
- jonursenbach 14y ago
  I've been a very happy Backblaze customer for over a year now. http://www.backblaze.com/ http://www.backblaze.com/
  - jrs235 14y ago
    If backblaze backs up my dropbox folder and this dropbox bug affects my files that are getting backedup, if I don't catch the 0 byte files within the 4 week backblaze revision window, won't I still lose my files?
    - antidaily 14y ago
      Yep. And your wife will divorce you.
    - majormajor 14y ago
      Crashplan hangs on to deleted files for quite some time. I just used it last weekend to recover some stuff I'd deleted about a year ago and realized I wanted back. I was pleasantly surprised to find them still in the system. It has a lot of options for retaining old versions, too: http://support.crashplan.com/doku.php/reference/version_retention_and_backup_frequency http://support.crashplan.com/doku.php/reference/version_rete...
uptown 14y ago
Dropbox for syncing. Crashplan for backing up. That combo hasn't failed me yet.
- bitcartel 14y ago
  I quite like Crashplan, but they have lost data for customers before. http://jeffreydonenfeld.com/blog/2011/12/crashplan-online-backup-lost-my-entire-backup-archive/ http://jeffreydonenfeld.com/blog/2011/12/crashplan-online-ba...
  - ejdyksen 14y ago
    PSA: with CrashPlan, you can simultaneously backup to your own server(s) or external hard drives right alongside their service. You can also backup to a friend's computer (this was actually their original business model--P2P backup).
    - usefulcat 14y ago
      Clarification: you can back up to your own servers iff said servers are also running Crashplan. What you can't do, unfortunately, is back up to an ordinary shared folder.
      - ejdyksen 14y ago
        Further clarification: this is correct, but (if someone is reading this and thinking about doing it), you can install the free version of CrashPlan and it works just fine as a backup server.
        Dylan16807 14y ago
        It's kind of a ram hog, though.
- ed209 14y ago
  I'd love to use CrashPlan for the bulk of my older projects but I have 350GB data and as I'm from the UK there is no way to seed it. It would take me something like a month to upload all that data.
Splines 14y ago
My content is fine, I have about 5.5 GB of content, and usually push content using the Windows client.
bitcartel 14y ago
Although Dropbox isn't a backup service, it offers file versioning, so you would think you could recover from such a situation. Google Drive also offers file versioning, but the devil is in the detail - they do delete older versions: https://support.google.com/drive/bin/answer.py?hl=en&answer=2375120 https://support.google.com/drive/bin/answer.py?hl=en&ans... For RAID like protection with Dropbox and other providers, you can roll your own BRIC (Redundant Bunch of Independent Clouds). I did this using Tahoe-LAFS to stripe data across storage providers. Requires a bit of set up, has some caveats, but does work. If you use with duplicity you have versioning on top of a distributed, encrypted, redundant store. http://news.ycombinator.com/item?id=4689238 http://news.ycombinator.com/item?id=4689238
- konklone 14y ago
  Part of what makes this such a bad bug is because Dropbox's file versioning is suffering from it. I have file versioning enabled, but not all of my files were able to use it, because their pre-0-byte history had been wiped.
deleted 14y ago
[deleted]
goostavos 14y ago
How many of you is this affecting? I'd be curious to know how many keep the only copy of their file on dropbox, or any cloud service for that matter. I've never trusted any service enough to be solely responsibly for my important files. I use Dropbox primarily as a tool to synch content between my desktop/laptop/phone, but any significant change I make to those files gets saved locally 100% of the time. I am not a very trusting man.
- fady 14y ago
  2 files here. it did delete a boilerplate site i was working on...thank goodness i backed it up locally.
- pseudonym 14y ago
  I think I have a couple files affected, but I can't tell for sure--thankfully, none of them are critical, and may just be generated as temp files. That said, I'm definitely writing a script to do nightly backups of the contents of my dropbox folder going forward.
  - mitchty 14y ago
    Just use something like rsnapshot http://www.rsnapshot.org/ http://www.rsnapshot.org/
- piyush_soni 14y ago
  Don't do this. I had some critical files which this stupid Dropbox sync system deleted without asking because I had removed the local copy of the folder on my machine. I couldn't even find a simple setting there, which says "don't delete files while syncing", or even something like "warn before deleting files". It totally sucks.
  - andreasvc 14y ago
    I'm afraid you have simply failed to grasp the meaning of syncing. The whole point of Dropbox is to have a folder which always has exactly the same contents on all your Dropbox installations, and the copy on their servers.
tnorthcutt 14y ago
Wow. 15 Files affected here. I haven't checked to see if any of them are unrecoverable (none of them are vital), but this does seem like a very bad bug.
- andreasvc 14y ago
  Are you sure those files were actually truncated? Because the command will simply show files of 0 bytes, which is not necessarily an indication of a bug, only if the file was supposed to contain something.
joejohnson 14y ago
This guy didn't backup his computer before upgrading the OS? That was his first mistake.
- konklone 14y ago
  I backed up everything else. :) I usually back up my Dropbox just in case, too, but I just didn't this time. I won't make that mistake again.
  - kwijibob 14y ago
    I do this all the time. It is the strength of dropbox that I can lean on it to do a fresh install of the OS and simply get my files back via DB.
alttab 14y ago
Well, ... it is DROPbox...?
iceron 14y ago
FYI: I'm pretty sure Windows users can navigate to their Dropbox folder and type 0 bytes in the search bar to see if they were affected.
- nosecreek 14y ago
  I had to type `size: 0 bytes`
eli 14y ago
Huh. Is there an easy way to confirm files affected versus files that were intentionally set to zero bytes?
wissler 14y ago
At least one of my files was effected and not recoverable (hopefully the zero-size test is reliable).
- zalambar 14y ago
  0 byte files are not necessarily a sign of a bug. There are many cases where you, or the tools you use, may intentionally create 0 byte files which would appear in DropBox. There is a problem only if you see empty files which you expected to contain data.
  - wissler 14y ago
    Obviously...
  - wissler 14y ago
    I mean, I know there are idiots on the Internet, but at some point your assumption that someone is an idiot is idiotic. I know where this file came from and I haven't touched it. Dropbox hammered it.
    - zalambar 14y ago
      Take it easy, I wasn't trying insult you or suggest that I believe you don't know what you are doing. I read "hopefully the zero-size test is reliable" as "is a reliable way to detect if there is a problem with my files" and that is what I was trying to comment on. Apparently I misunderstood.
larme 14y ago
For all the people saying you need a local backup, in this case you not only need to keep a local backup, but also it have to be a versioned one.
- bruceboughton 14y ago
  Backup isn't backup unless it's versioned. Otherwise, how do you restore that file you accidentally deleted a few days ago?
bloaf 14y ago
Remember, cloud storage doesn't count as a backup if its the only place you've stored your files.
mikegirouard 14y ago
> 2 other files (precious family photos) were also affected, but it happened recently enough to be recovered manually by Dropbox engineers. It's awful that it had to come to that, but it's reassuring that they will be willing to work with you on that level.
whichdan 14y ago
How viable would it be to keep all of your data in a git repository? Let's say I'm backing up 25gb of music, could I have Dropbox sync everything but the .git folder, and just do a git revert if shit goes down? Will it end up using twice as much space?
- rmc 14y ago
  Look at git-annex, it works on top of git and is good at handling large files, and will not require twice the space. If you use plain git, you'll probably need twice the space. Files stored in git are compressed, but music is already compressed, so there will be little savings. You'll also need a lot of memory if you want to copy files VIA git.
  - whichdan 14y ago
    Thanks for the suggestion - it also led me to some other cool projects like git-media. If git doesn't keep its own copy of the data, a Dropbox "failure" would still be unrecoverable, right?
    - rmc 14y ago
      If git doesn't keep its own copy of the data Plain regular git does keep a copy of it's data in your .git folder, and every checkout/clone/copy of that repository stores the data, and it stores all old copies of all files. That's how git works. It also makes it a bit unweildy for large files like that. What's cool (about git) is that the hash revisions (i.e. what git uses instead of version numbers) is basically a checksum of every file and every old version of every file. So if an old version of a file changed, the checksum would change and you'd be on a different branch!
      - whichdan 14y ago
        That makes a lot of sense, thanks!
intellegacy 14y ago
Happened to my mother and her friend a week ago. Only they lost all their files. They weren't permanently unrecoverable, thankfully, only deleted. The trust my mother had in dropbox is now gone, and probably will remain so for the next couple years.
- keithpeter 14y ago
  As an end user myself on Linux, I was worried to read this. I keep teaching materials in the Dropbox folder so only a small file size, but they are important. As mentioned below at http://news.ycombinator.com/item?id=4704176 http://news.ycombinator.com/item?id=4704176 I have developed the bad habit of editing files directly in Dropbox. I may need to revisit this and use Dropbox as a sync only, edit elsewhere and copy over. The command find /home/keith/Dropbox -size 0 shows only cache files for deleted files, and some backup files that I saved while empty (I know those should be zero bytes). A personal work around is a simple bash script to copy Dropbox directory to another with the date as directory name, I'm running this once a day or so. Then my normal old-school backup onto an external drive will catch each day's dropbox. Surprising how convenient I found automatic file sync, and how quickly I came to trust the dropbox daemon running in the background on 3 computers!
apike 14y ago
I have all my data, media, and documents in Dropbox (80GB) and while I have 11255 zero-byte files, none of them are likely Dropbox's fault. Most of them are empty logfiles, .svn and .git noise in old projects, and the like.
M4v3R 14y ago
Isn't this related to Ext4 bug that was found recently? [1] [1] http://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ http://www.phoronix.com/scan.php?page=news_item&px=MTIxN...
piyush_soni 14y ago
Dropbox has lost my files as well. I had kept some very important private files there which I just wanted to keep there without the need of updating them/syncing them. After uploading, I removed the local copy of that private top level folder, as I didn't want to keep a local copy in my computer. With the next sync, this STUPID SH*T removed the folder from their server as well, without any warning of any sort! Any system which deletes the files without asking, should just not be used or depended on.
- nullymcnull 14y ago
  You deleted the folder, and dropbox synced the deletion. This is expected behaviour. I don't know why you entrusted crucial private files to a service you didn't fully understand.
- djrogers 14y ago
  It sounds like you don't quite understand how Dropbox is supposed to work - it replicates whatever you have locally to the cloud and back down to your other machines. Obviously with a design like that, deleting files from your local dropbox would delete them from other machines. This isn't a case of dropbox losing your files, it's a case of you not understanding how the tool was designed and how it works.
  - piyush_soni 14y ago
    Many sync software offer multiple types of sync (e.g. MS SyncToy), and many are extremely cautious before deleting multiple files. Dropbox didn't do that. Yes, I failed to totally understand how did it work, as I deleted one of the root level folders and expected it'd not sync a folder which is not there on the local machine, but one of the duties of software is to prevent or try to reduce common human errors.
    - Dylan16807 14y ago
      Dropbox is way simpler than that. It syncs precisely one folder. There is really no way for it to not sync file deletions without making a massive clutter.
mech4bg 14y ago
It surprises me that someone would rely entirely on dropbox as their backup tool AND primary file location, especially when it happy manipulates the files locally. I backup to an external HDD and to the cloud and still have the originals (as well as having extra copies again of my music and photos synced across my computers) - the more redundancy you have the better. It sucks that so many people need bad stuff to happen to them to do something about it - I'm so thankful that storage became so cheap before anything really catastrophic happened to me. I've lost data in the past but it was back when so many things were offline, nowadays it's CRITICAL to have a good backup plan.
- konklone 14y ago
  I didn't go into detail on my backup strategy in the post, but it's (slightly) better than that. I only use Dropbox for media - music, images, photos. Some if it is irreplaceable, but none of it is life-crucial. That stuff is backed up separately, and copied to private space on my web server.
  - mech4bg 14y ago
    I think I consider photos to be some of the most important stuff to be backed up... obviously you can live without them, but as you said, some photos are irreplaceable, and can hold so much meaning. At least with Facebook / Picasa Web Albums many of the most important photos have been backed up (potentially at a lower quality). To be fair I never would have considered such a bug when using dropbox - I would probably have considered it safe considering you have a local copy of your data, especially since as other pointed out they present themselves as a backup service. I almost got hit by this locally actually, I synchronize folders against multiple PCs, and the exact same thing happened, and a number of files had their bytes zeroed out and then this was propagated through the network. Thankfully I spotted it before all copies were overwritten and fixed it, but that's where you also want something like Time Machine. Ugh, so many ways to lose data, even when you're doing the 'right' thing!
    - konklone 14y ago
      You're right, photos can be irreplaceable and I'm probably going to change how I back them up now. My choice of what is and isn't in Dropbox isn't actually based on importance -- but on what I both want on my laptop, but would want to restore if the laptop were stolen. Kind of weird criteria, but it (at one time) made sense to me.
duncans 14y ago
If you're on windows the following Powershell may help diagnose if you've been affected: gci $env:USERPROFILE\Dropbox -r | where { $_.Length -eq 0 }
matt_holden 14y ago
(Hi all, I'm the PM for the Dropbox desktop client team.) I just wanted to let you all know that we take any claims like this really seriously. There aren't any known bugs on the Dropbox side that would cause this, and unfortunately there are potential causes such as hardware errors, filesystem corruption, and other OS issues (including those like http://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ http://www.phoronix.com/scan.php?page=news_item&px=MTIxN... which another poster pointed out) that can corrupt data or create zero byte files. Nevertheless we will continue to look into this just to be sure, and we also work hard to find ways for Dropbox to shield users even when the OS, disk or other components fail (our undelete/file revisions and Packrat are among these).
- konklone 14y ago
  Matt, Thanks for posting here. The reason I'm confident that this bug is not due to filesystem errors on my machine's part is because Dropbox's version history for these files has become corrupted. If you read over the details and correspondence, you'll see that Dropbox engineers were able to recover two of my files that Dropbox's version history had reported as only having a 0-byte version. These files were edited recently, within the 30-day window that all Dropbox users, Packrat or not, have version control for. The files had clearly been whole when first synced to Dropbox, but that version was not listed in Dropbox's history, and so I had no power to restore them. Even if my own disk had spontaneously 0-byte'd those files, this should not have caused Dropbox to lose the ability to restore it to its original version. More circumstantially, others are reporting similar issues: https://twitter.com/dangillmor/status/261921738441515009 https://twitter.com/dangillmor/status/261921738441515009 - https://twitter.com/pc1oad1etter/status/261957001234505728 https://twitter.com/pc1oad1etter/status/261957001234505728 - https://twitter.com/frr149/status/261957708746469378 https://twitter.com/frr149/status/261957708746469378 - http://news.ycombinator.com/item?id=4704236 http://news.ycombinator.com/item?id=4704236 - http://news.ycombinator.com/item?id=4704178 http://news.ycombinator.com/item?id=4704178 - http://news.ycombinator.com/item?id=4704063 http://news.ycombinator.com/item?id=4704063 - http://news.ycombinator.com/item?id=4704485 http://news.ycombinator.com/item?id=4704485 It's tough to tell from these small updates whether these users had their files 0-byte'd only locally, possibly by the ext4 bug, or whether they've also verified that it's not recoverable from Dropbox. In addressing this bug report, please specifically address Dropbox's loss of version history. The two files that Dropbox engineers recovered had their version history wiped within the 30 day window.
gcv 14y ago
No problems on my system, but I make a point of backing up everything sensitive to an external hard drive occasionally, and using Arq to do hourly backups to S3.
ssebro 14y ago
I've had a similar thing happen to me, but I was able use a python script to pull the (hidden, old) files from a cache on my machine. http://www.dropboxwiki.com/TipsAndTricks/RestoreFilesAndDirectories http://www.dropboxwiki.com/TipsAndTricks/RestoreFilesAndDire...
acabal 14y ago
Never trust anyone but yourself with valuable files like family photos. I don't care if they promise 110% uptime and reliability. The files you really care about, in the very practical end, are your own responsibility regardless of how much money you pay to other people. You can sue them till you're blue but it won't get your files back. To that extent I keep my important files backed up not just in Dropbox, but also to Crashplan, and to a spinning-rust hard drive especially kept for backups that I protect in my home. That's three points of failure I can recover from if something goes wrong; and if all three fail at once, then I probably have worse things to worry about, like the zombie apocalypse.
- Bud 14y ago
  If you really care about photos, you will print them out, and put them in a photo album, and print them again, and put them in a safe deposit box or mail them to some relatives.
- andrewljohnson 14y ago
  The last person I trust with valuable files like family photos is myself. I am a bad sysadmin, bad at back-ups, bad at security, bad at redundancy. And I would guess that describes 99.99% of people who care about family photos.
  - acabal 14y ago
    Well, that's why I trust two other companies, plus myself. I don't know, doing basic backups aren't super-hard for someone who's reading HN. My setup is really just a regular Linux box with a 1TB HDD sitting in a closet of mine, with dynamic dns pointing to my own domain (not even a strict necessity), and I rsync it whenever I have new data like photos. That box in turn syncs to Crashplan and Dropbox, which is automatic. Yeah rsync and the concept of a backup PC is beyond mere mortals, but for those of us here--if Dropbox (or any service) is your only backup, I think you can only blame yourself.
mebesilly 14y ago
That is why I use Spideroak for remote backup and use Dropbox only for syncing files. Any thing important is also backed locally and at yet another remote site owned by me. I do not really trust Spideroak to not eat my files either.
ChuckMcM 14y ago
Thinking about how dropbox might be implemented I wonder if the following series of events is possible: 1) Process which syncs files to the server fails to get access to a local fils, sees it as length 0 2) Process proceeds to tell the server file is length 0 and the server updates the file to be 0 length. 3) Access to the local file is restored and client notices that the server file is 'newer' than the local client version and it was length zero so it truncates the client copy to length 0. The thing is, I can imagine a number of scenarios where the local file might seem to be zero length (oplocks on NTFS volumes being one)
shangaslammi 14y ago
I've also had files zeroed out on a few occasions. I run Dropbox over a wide variety of machines and while this is completely anecdotal, the zero update has always come from one of my Ubuntu machines, which leads me to believe that this is specific to the Linux version of Dropbox. Also, I had several occurrences throughout 2011 but so far hadn't lost anything during 2012, so either something has been fixed or I've just been lucky. :) So, if you've noticed zero length files lately, can you check the timestamp on the last update? Is it recent or over a year ago? You can also go to that date in your event log in the Dropbox web UI and see what was happening around that time.
mike-cardwell 14y ago
Just added the following to the crontab on my laptop so when it boots up, it emails me about any zero byte files. # Look for zero byte files in Dropbox MAILTO=my.email.address@example.com @reboot /usr/bin/find ~/Dropbox/ -type f -size 0 I also have full system backups going back three months, with two hourly incremental backups for the last two. So if I get an email about any zero byte files, I should be fine restoring them from my own backups.
daniel-cussen 14y ago
Everything can.
hnriot 14y ago
Wouldn't this seem like something that can be easily solved. Dropbox or any rsync+rcs scheme will never destroy a file unless either one endpoint of the sync has truncated the file or there is a file system corruption. In the case of the former the rcs portion will take care of things, restore the pre-truncated file and have that sync'd around. In the case of a file system corruption Dropbox should be doing file integrity checks. For example some simple things that spring to mind, most file types b&w e built in signatures, header blocks, that validate the files extension type, if the file fails to "look like" the claimed extension type then don't sync it, it flag it for user attention. I do Dropbox manually, I rsync a list of folders to a linode instance running mercurial. It's simpler, scriptable, more flexible and as fail safe as Dropbox. If something does get corrupted its really simple to go back in the version history. I can't remember the last time I had a file system corruption with ext3, I suppose they still happen, but not to me in solid use for years. Obviously my mercurial repository is also backed up regularly. I don't agree that a backup system has to be restore tested periodically, instead I believe that the restore process has to be an integral part of the workflow. In my scenario, I rsync to hg, commit and push, then from other machines in my workflow (or more likely vm instances) I pull+rsync back. This way the backup and restore cycles are just part if the workflow and everything is version controlled at the same time.
nosecreek 14y ago
Can someone clarify the pricing for Dropbox's packrat? Is it free for pro users and free users have option to pay 3.99/month to get it, or do you have to be a pro user and then you can pay $3.99/month to get it? I think its the later, but I'm finding the Dropbox help page for it really unclear (sounds like it is free for all pro users).
wikwocket 14y ago
If you're on Windows and want to quickly check if you have 0-byte files in your Dropbox, open the Dropbox folder in Explorer and type "size:empty" in the search box, and hit enter. It will show any 0-byte files. If you'd like a batch file to check this, that you can run periodically (perhaps programmatically), try this: @echo off for /r C:\Users\username\Dropbox %%F in (*.*) do (if %%~zF LSS 1 echo %%F) pause If you'd like to echo that list to a file, change the middle line to: (for /r C:\Users\username\Dropbox %%F in (*.*) do (if %%~zF LSS 1 echo %%F)) > C:\Temp\emptyfiles.txt

193 points by joshuacc 14y ago

c16 14y ago

Yep, I'm affected by this bug too.. Thankfully nothing important, but makes me wonder if I should leave my truecrypt folder on DropBox.

gte910h 14y ago
You should both distribute your file you want to keep to several devices, as well as a service who's job it is to back things up. Dropbox is only the former.
- c16 14y ago
  I have a backup on a USB, but updating that daily is a hassle hence why I used DropBox- their added 2 step auth was also a big bonus, but if I lose that file I'll have a hard time getting my passwords back.. As jpadvo said, I might look into making some one way backups to S3 or something.
  - gte910h 14y ago
    Buy a time capsule and back it up locally or get a subscription to something like crash plan pro.

jpadvo 14y ago

It's always important to remember the difference between a syncing service and a backup service. A syncing service sometimes feels like a backup, because you can use it to recover files if a local device is destroyed or lost. HOWEVER, any service capable of syncing files is equally capable of destroying them. It's important to have an automated one-way backup system that you can manually restore from. Something like Tarsnap [1] looks like a really good possibility (I haven't used it myself, but it seems solid) [1] http://www.tarsnap.com/gettingstarted.html http://www.tarsnap.com/gettingstarted.html

erichocean 14y ago
HOWEVER, any service capable of syncing files is equally capable of destroying them. That's simply not true (I've built sync systems that are incapable of destroying files). IMO, syncing designs that do have unrevokable overwrites are inherently brittle. (I don't know if Dropbox is built that way, but AFIAK, iCloud (and MobileMe before it) is -- and it sucks.)
- Dylan16807 14y ago
  Dropbox is revocable, the issue is for how long. In addition to online versions, the client holds on to old versions for three days. It used to be longer but the cache could get unreasonably large.
jonursenbach 14y ago
I've been a very happy Backblaze customer for over a year now. http://www.backblaze.com/ http://www.backblaze.com/
- jrs235 14y ago
  If backblaze backs up my dropbox folder and this dropbox bug affects my files that are getting backedup, if I don't catch the 0 byte files within the 4 week backblaze revision window, won't I still lose my files?
  - antidaily 14y ago
    Yep. And your wife will divorce you.
  - majormajor 14y ago
    Crashplan hangs on to deleted files for quite some time. I just used it last weekend to recover some stuff I'd deleted about a year ago and realized I wanted back. I was pleasantly surprised to find them still in the system. It has a lot of options for retaining old versions, too: http://support.crashplan.com/doku.php/reference/version_retention_and_backup_frequency http://support.crashplan.com/doku.php/reference/version_rete...

uptown 14y ago

Dropbox for syncing. Crashplan for backing up. That combo hasn't failed me yet.

bitcartel 14y ago
I quite like Crashplan, but they have lost data for customers before. http://jeffreydonenfeld.com/blog/2011/12/crashplan-online-backup-lost-my-entire-backup-archive/ http://jeffreydonenfeld.com/blog/2011/12/crashplan-online-ba...
- ejdyksen 14y ago
  PSA: with CrashPlan, you can simultaneously backup to your own server(s) or external hard drives right alongside their service. You can also backup to a friend's computer (this was actually their original business model--P2P backup).
  - usefulcat 14y ago
    Clarification: you can back up to your own servers iff said servers are also running Crashplan. What you can't do, unfortunately, is back up to an ordinary shared folder.
    - ejdyksen 14y ago
      Further clarification: this is correct, but (if someone is reading this and thinking about doing it), you can install the free version of CrashPlan and it works just fine as a backup server.
      - Dylan16807 14y ago
        It's kind of a ram hog, though.
ed209 14y ago
I'd love to use CrashPlan for the bulk of my older projects but I have 350GB data and as I'm from the UK there is no way to seed it. It would take me something like a month to upload all that data.

Splines 14y ago

My content is fine, I have about 5.5 GB of content, and usually push content using the Windows client.

bitcartel 14y ago

Although Dropbox isn't a backup service, it offers file versioning, so you would think you could recover from such a situation. Google Drive also offers file versioning, but the devil is in the detail - they do delete older versions: https://support.google.com/drive/bin/answer.py?hl=en&answer=2375120 https://support.google.com/drive/bin/answer.py?hl=en&ans... For RAID like protection with Dropbox and other providers, you can roll your own BRIC (Redundant Bunch of Independent Clouds). I did this using Tahoe-LAFS to stripe data across storage providers. Requires a bit of set up, has some caveats, but does work. If you use with duplicity you have versioning on top of a distributed, encrypted, redundant store. http://news.ycombinator.com/item?id=4689238 http://news.ycombinator.com/item?id=4689238

konklone 14y ago
Part of what makes this such a bad bug is because Dropbox's file versioning is suffering from it. I have file versioning enabled, but not all of my files were able to use it, because their pre-0-byte history had been wiped.

deleted 14y ago

[deleted]

goostavos 14y ago

How many of you is this affecting? I'd be curious to know how many keep the only copy of their file on dropbox, or any cloud service for that matter. I've never trusted any service enough to be solely responsibly for my important files. I use Dropbox primarily as a tool to synch content between my desktop/laptop/phone, but any significant change I make to those files gets saved locally 100% of the time. I am not a very trusting man.

fady 14y ago
2 files here. it did delete a boilerplate site i was working on...thank goodness i backed it up locally.
pseudonym 14y ago
I think I have a couple files affected, but I can't tell for sure--thankfully, none of them are critical, and may just be generated as temp files. That said, I'm definitely writing a script to do nightly backups of the contents of my dropbox folder going forward.
- mitchty 14y ago
  Just use something like rsnapshot http://www.rsnapshot.org/ http://www.rsnapshot.org/
piyush_soni 14y ago
Don't do this. I had some critical files which this stupid Dropbox sync system deleted without asking because I had removed the local copy of the folder on my machine. I couldn't even find a simple setting there, which says "don't delete files while syncing", or even something like "warn before deleting files". It totally sucks.
- andreasvc 14y ago
  I'm afraid you have simply failed to grasp the meaning of syncing. The whole point of Dropbox is to have a folder which always has exactly the same contents on all your Dropbox installations, and the copy on their servers.

tnorthcutt 14y ago

Wow. 15 Files affected here. I haven't checked to see if any of them are unrecoverable (none of them are vital), but this does seem like a very bad bug.

andreasvc 14y ago
Are you sure those files were actually truncated? Because the command will simply show files of 0 bytes, which is not necessarily an indication of a bug, only if the file was supposed to contain something.

joejohnson 14y ago

This guy didn't backup his computer before upgrading the OS? That was his first mistake.

konklone 14y ago
I backed up everything else. :) I usually back up my Dropbox just in case, too, but I just didn't this time. I won't make that mistake again.
- kwijibob 14y ago
  I do this all the time. It is the strength of dropbox that I can lean on it to do a fresh install of the OS and simply get my files back via DB.

alttab 14y ago

Well, ... it is DROPbox...?

iceron 14y ago

FYI: I'm pretty sure Windows users can navigate to their Dropbox folder and type 0 bytes in the search bar to see if they were affected.

nosecreek 14y ago
I had to type `size: 0 bytes`

eli 14y ago

Huh. Is there an easy way to confirm files affected versus files that were intentionally set to zero bytes?

wissler 14y ago

At least one of my files was effected and not recoverable (hopefully the zero-size test is reliable).

zalambar 14y ago
0 byte files are not necessarily a sign of a bug. There are many cases where you, or the tools you use, may intentionally create 0 byte files which would appear in DropBox. There is a problem only if you see empty files which you expected to contain data.
- wissler 14y ago
  Obviously...
- wissler 14y ago
  I mean, I know there are idiots on the Internet, but at some point your assumption that someone is an idiot is idiotic. I know where this file came from and I haven't touched it. Dropbox hammered it.
  - zalambar 14y ago
    Take it easy, I wasn't trying insult you or suggest that I believe you don't know what you are doing. I read "hopefully the zero-size test is reliable" as "is a reliable way to detect if there is a problem with my files" and that is what I was trying to comment on. Apparently I misunderstood.

larme 14y ago

For all the people saying you need a local backup, in this case you not only need to keep a local backup, but also it have to be a versioned one.

bruceboughton 14y ago
Backup isn't backup unless it's versioned. Otherwise, how do you restore that file you accidentally deleted a few days ago?

bloaf 14y ago

Remember, cloud storage doesn't count as a backup if its the only place you've stored your files.

mikegirouard 14y ago

> 2 other files (precious family photos) were also affected, but it happened recently enough to be recovered manually by Dropbox engineers. It's awful that it had to come to that, but it's reassuring that they will be willing to work with you on that level.

whichdan 14y ago

How viable would it be to keep all of your data in a git repository? Let's say I'm backing up 25gb of music, could I have Dropbox sync everything but the .git folder, and just do a git revert if shit goes down? Will it end up using twice as much space?

rmc 14y ago
Look at git-annex, it works on top of git and is good at handling large files, and will not require twice the space. If you use plain git, you'll probably need twice the space. Files stored in git are compressed, but music is already compressed, so there will be little savings. You'll also need a lot of memory if you want to copy files VIA git.
- whichdan 14y ago
  Thanks for the suggestion - it also led me to some other cool projects like git-media. If git doesn't keep its own copy of the data, a Dropbox "failure" would still be unrecoverable, right?
  - rmc 14y ago
    If git doesn't keep its own copy of the data Plain regular git does keep a copy of it's data in your .git folder, and every checkout/clone/copy of that repository stores the data, and it stores all old copies of all files. That's how git works. It also makes it a bit unweildy for large files like that. What's cool (about git) is that the hash revisions (i.e. what git uses instead of version numbers) is basically a checksum of every file and every old version of every file. So if an old version of a file changed, the checksum would change and you'd be on a different branch!
    - whichdan 14y ago
      That makes a lot of sense, thanks!

intellegacy 14y ago

Happened to my mother and her friend a week ago. Only they lost all their files. They weren't permanently unrecoverable, thankfully, only deleted. The trust my mother had in dropbox is now gone, and probably will remain so for the next couple years.

keithpeter 14y ago
As an end user myself on Linux, I was worried to read this. I keep teaching materials in the Dropbox folder so only a small file size, but they are important. As mentioned below at http://news.ycombinator.com/item?id=4704176 http://news.ycombinator.com/item?id=4704176 I have developed the bad habit of editing files directly in Dropbox. I may need to revisit this and use Dropbox as a sync only, edit elsewhere and copy over. The command find /home/keith/Dropbox -size 0 shows only cache files for deleted files, and some backup files that I saved while empty (I know those should be zero bytes). A personal work around is a simple bash script to copy Dropbox directory to another with the date as directory name, I'm running this once a day or so. Then my normal old-school backup onto an external drive will catch each day's dropbox. Surprising how convenient I found automatic file sync, and how quickly I came to trust the dropbox daemon running in the background on 3 computers!

apike 14y ago

I have all my data, media, and documents in Dropbox (80GB) and while I have 11255 zero-byte files, none of them are likely Dropbox's fault. Most of them are empty logfiles, .svn and .git noise in old projects, and the like.

M4v3R 14y ago

Isn't this related to Ext4 bug that was found recently? [1] [1] http://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ http://www.phoronix.com/scan.php?page=news_item&px=MTIxN...

piyush_soni 14y ago

Dropbox has lost my files as well. I had kept some very important private files there which I just wanted to keep there without the need of updating them/syncing them. After uploading, I removed the local copy of that private top level folder, as I didn't want to keep a local copy in my computer. With the next sync, this STUPID SH*T removed the folder from their server as well, without any warning of any sort! Any system which deletes the files without asking, should just not be used or depended on.

nullymcnull 14y ago
You deleted the folder, and dropbox synced the deletion. This is expected behaviour. I don't know why you entrusted crucial private files to a service you didn't fully understand.
djrogers 14y ago
It sounds like you don't quite understand how Dropbox is supposed to work - it replicates whatever you have locally to the cloud and back down to your other machines. Obviously with a design like that, deleting files from your local dropbox would delete them from other machines. This isn't a case of dropbox losing your files, it's a case of you not understanding how the tool was designed and how it works.
- piyush_soni 14y ago
  Many sync software offer multiple types of sync (e.g. MS SyncToy), and many are extremely cautious before deleting multiple files. Dropbox didn't do that. Yes, I failed to totally understand how did it work, as I deleted one of the root level folders and expected it'd not sync a folder which is not there on the local machine, but one of the duties of software is to prevent or try to reduce common human errors.
  - Dylan16807 14y ago
    Dropbox is way simpler than that. It syncs precisely one folder. There is really no way for it to not sync file deletions without making a massive clutter.

mech4bg 14y ago

It surprises me that someone would rely entirely on dropbox as their backup tool AND primary file location, especially when it happy manipulates the files locally. I backup to an external HDD and to the cloud and still have the originals (as well as having extra copies again of my music and photos synced across my computers) - the more redundancy you have the better. It sucks that so many people need bad stuff to happen to them to do something about it - I'm so thankful that storage became so cheap before anything really catastrophic happened to me. I've lost data in the past but it was back when so many things were offline, nowadays it's CRITICAL to have a good backup plan.

konklone 14y ago
I didn't go into detail on my backup strategy in the post, but it's (slightly) better than that. I only use Dropbox for media - music, images, photos. Some if it is irreplaceable, but none of it is life-crucial. That stuff is backed up separately, and copied to private space on my web server.
- mech4bg 14y ago
  I think I consider photos to be some of the most important stuff to be backed up... obviously you can live without them, but as you said, some photos are irreplaceable, and can hold so much meaning. At least with Facebook / Picasa Web Albums many of the most important photos have been backed up (potentially at a lower quality). To be fair I never would have considered such a bug when using dropbox - I would probably have considered it safe considering you have a local copy of your data, especially since as other pointed out they present themselves as a backup service. I almost got hit by this locally actually, I synchronize folders against multiple PCs, and the exact same thing happened, and a number of files had their bytes zeroed out and then this was propagated through the network. Thankfully I spotted it before all copies were overwritten and fixed it, but that's where you also want something like Time Machine. Ugh, so many ways to lose data, even when you're doing the 'right' thing!
  - konklone 14y ago
    You're right, photos can be irreplaceable and I'm probably going to change how I back them up now. My choice of what is and isn't in Dropbox isn't actually based on importance -- but on what I both want on my laptop, but would want to restore if the laptop were stolen. Kind of weird criteria, but it (at one time) made sense to me.

duncans 14y ago

If you're on windows the following Powershell may help diagnose if you've been affected: gci $env:USERPROFILE\Dropbox -r | where { $_.Length -eq 0 }

matt_holden 14y ago

(Hi all, I'm the PM for the Dropbox desktop client team.) I just wanted to let you all know that we take any claims like this really seriously. There aren't any known bugs on the Dropbox side that would cause this, and unfortunately there are potential causes such as hardware errors, filesystem corruption, and other OS issues (including those like http://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ http://www.phoronix.com/scan.php?page=news_item&px=MTIxN... which another poster pointed out) that can corrupt data or create zero byte files. Nevertheless we will continue to look into this just to be sure, and we also work hard to find ways for Dropbox to shield users even when the OS, disk or other components fail (our undelete/file revisions and Packrat are among these).

konklone 14y ago
Matt, Thanks for posting here. The reason I'm confident that this bug is not due to filesystem errors on my machine's part is because Dropbox's version history for these files has become corrupted. If you read over the details and correspondence, you'll see that Dropbox engineers were able to recover two of my files that Dropbox's version history had reported as only having a 0-byte version. These files were edited recently, within the 30-day window that all Dropbox users, Packrat or not, have version control for. The files had clearly been whole when first synced to Dropbox, but that version was not listed in Dropbox's history, and so I had no power to restore them. Even if my own disk had spontaneously 0-byte'd those files, this should not have caused Dropbox to lose the ability to restore it to its original version. More circumstantially, others are reporting similar issues: https://twitter.com/dangillmor/status/261921738441515009 https://twitter.com/dangillmor/status/261921738441515009 - https://twitter.com/pc1oad1etter/status/261957001234505728 https://twitter.com/pc1oad1etter/status/261957001234505728 - https://twitter.com/frr149/status/261957708746469378 https://twitter.com/frr149/status/261957708746469378 - http://news.ycombinator.com/item?id=4704236 http://news.ycombinator.com/item?id=4704236 - http://news.ycombinator.com/item?id=4704178 http://news.ycombinator.com/item?id=4704178 - http://news.ycombinator.com/item?id=4704063 http://news.ycombinator.com/item?id=4704063 - http://news.ycombinator.com/item?id=4704485 http://news.ycombinator.com/item?id=4704485 It's tough to tell from these small updates whether these users had their files 0-byte'd only locally, possibly by the ext4 bug, or whether they've also verified that it's not recoverable from Dropbox. In addressing this bug report, please specifically address Dropbox's loss of version history. The two files that Dropbox engineers recovered had their version history wiped within the 30 day window.

gcv 14y ago

No problems on my system, but I make a point of backing up everything sensitive to an external hard drive occasionally, and using Arq to do hourly backups to S3.

ssebro 14y ago

I've had a similar thing happen to me, but I was able use a python script to pull the (hidden, old) files from a cache on my machine. http://www.dropboxwiki.com/TipsAndTricks/RestoreFilesAndDirectories http://www.dropboxwiki.com/TipsAndTricks/RestoreFilesAndDire...

acabal 14y ago

Never trust anyone but yourself with valuable files like family photos. I don't care if they promise 110% uptime and reliability. The files you really care about, in the very practical end, are your own responsibility regardless of how much money you pay to other people. You can sue them till you're blue but it won't get your files back. To that extent I keep my important files backed up not just in Dropbox, but also to Crashplan, and to a spinning-rust hard drive especially kept for backups that I protect in my home. That's three points of failure I can recover from if something goes wrong; and if all three fail at once, then I probably have worse things to worry about, like the zombie apocalypse.

Bud 14y ago
If you really care about photos, you will print them out, and put them in a photo album, and print them again, and put them in a safe deposit box or mail them to some relatives.
andrewljohnson 14y ago
The last person I trust with valuable files like family photos is myself. I am a bad sysadmin, bad at back-ups, bad at security, bad at redundancy. And I would guess that describes 99.99% of people who care about family photos.
- acabal 14y ago
  Well, that's why I trust two other companies, plus myself. I don't know, doing basic backups aren't super-hard for someone who's reading HN. My setup is really just a regular Linux box with a 1TB HDD sitting in a closet of mine, with dynamic dns pointing to my own domain (not even a strict necessity), and I rsync it whenever I have new data like photos. That box in turn syncs to Crashplan and Dropbox, which is automatic. Yeah rsync and the concept of a backup PC is beyond mere mortals, but for those of us here--if Dropbox (or any service) is your only backup, I think you can only blame yourself.

mebesilly 14y ago

That is why I use Spideroak for remote backup and use Dropbox only for syncing files. Any thing important is also backed locally and at yet another remote site owned by me. I do not really trust Spideroak to not eat my files either.

ChuckMcM 14y ago

Thinking about how dropbox might be implemented I wonder if the following series of events is possible: 1) Process which syncs files to the server fails to get access to a local fils, sees it as length 0 2) Process proceeds to tell the server file is length 0 and the server updates the file to be 0 length. 3) Access to the local file is restored and client notices that the server file is 'newer' than the local client version and it was length zero so it truncates the client copy to length 0. The thing is, I can imagine a number of scenarios where the local file might seem to be zero length (oplocks on NTFS volumes being one)

shangaslammi 14y ago

I've also had files zeroed out on a few occasions. I run Dropbox over a wide variety of machines and while this is completely anecdotal, the zero update has always come from one of my Ubuntu machines, which leads me to believe that this is specific to the Linux version of Dropbox. Also, I had several occurrences throughout 2011 but so far hadn't lost anything during 2012, so either something has been fixed or I've just been lucky. :) So, if you've noticed zero length files lately, can you check the timestamp on the last update? Is it recent or over a year ago? You can also go to that date in your event log in the Dropbox web UI and see what was happening around that time.

mike-cardwell 14y ago

Just added the following to the crontab on my laptop so when it boots up, it emails me about any zero byte files. # Look for zero byte files in Dropbox MAILTO=my.email.address@example.com @reboot /usr/bin/find ~/Dropbox/ -type f -size 0 I also have full system backups going back three months, with two hourly incremental backups for the last two. So if I get an email about any zero byte files, I should be fine restoring them from my own backups.

daniel-cussen 14y ago

Everything can.

hnriot 14y ago

Wouldn't this seem like something that can be easily solved. Dropbox or any rsync+rcs scheme will never destroy a file unless either one endpoint of the sync has truncated the file or there is a file system corruption. In the case of the former the rcs portion will take care of things, restore the pre-truncated file and have that sync'd around. In the case of a file system corruption Dropbox should be doing file integrity checks. For example some simple things that spring to mind, most file types b&w e built in signatures, header blocks, that validate the files extension type, if the file fails to "look like" the claimed extension type then don't sync it, it flag it for user attention. I do Dropbox manually, I rsync a list of folders to a linode instance running mercurial. It's simpler, scriptable, more flexible and as fail safe as Dropbox. If something does get corrupted its really simple to go back in the version history. I can't remember the last time I had a file system corruption with ext3, I suppose they still happen, but not to me in solid use for years. Obviously my mercurial repository is also backed up regularly. I don't agree that a backup system has to be restore tested periodically, instead I believe that the restore process has to be an integral part of the workflow. In my scenario, I rsync to hg, commit and push, then from other machines in my workflow (or more likely vm instances) I pull+rsync back. This way the backup and restore cycles are just part if the workflow and everything is version controlled at the same time.

nosecreek 14y ago

Can someone clarify the pricing for Dropbox's packrat? Is it free for pro users and free users have option to pay 3.99/month to get it, or do you have to be a pro user and then you can pay $3.99/month to get it? I think its the later, but I'm finding the Dropbox help page for it really unclear (sounds like it is free for all pro users).

wikwocket 14y ago

If you're on Windows and want to quickly check if you have 0-byte files in your Dropbox, open the Dropbox folder in Explorer and type "size:empty" in the search box, and hit enter. It will show any 0-byte files. If you'd like a batch file to check this, that you can run periodically (perhaps programmatically), try this: @echo off for /r C:\Users\username\Dropbox %%F in (*.*) do (if %%~zF LSS 1 echo %%F) pause If you'd like to echo that list to a file, change the middle line to: (for /r C:\Users\username\Dropbox %%F in (*.*) do (if %%~zF LSS 1 echo %%F)) > C:\Temp\emptyfiles.txt