7 ms·
Ask HN: Is there a data loss bug lurking in MS365 backup solutions?
It sounds crazy, and maybe I'm just doing something dumb, but I've seen a similar issue in two different MS365 backup products this year. I can't reproduce it reliably, but feel like there could be a serious issue, even though I can't prove it.
My issue is specific to OneDrive. When I reconcile a backup set against live data, files are missing. I've had it happen with both Veeam Backup for MS365 and Synology Active Backup for MS365. Neither system reports any issues when backups run. I don't know the cause and I can't reproduce it, but it happens consistently for at least one of my tenants and seems to get worse over time. I've seen the issue on more than one tenant, so I don't think it's anything tenant specific and the only (untenable) solution I've come up with is to restart the backup from scratch.
The tenant that has the most issues is a business with about 200k files. The business owner owns the files and everyone else has access to a few shared folders near the top of the hierarchy. They have about 350GB of data which ends up being about 1TB of quota after versioning.
I originally ran into the issue with Veeam by randomly spot checking 1-2 files every once in a while and running into missing data by random chance. That made me realize I needed to do some kind of bulk reconciliation on a regular basis. I gave up on Veeam because they append the OneDrive file version to all restored files and it makes it difficult to reconcile. For example, locally restored files end up with ' (ver 2)' or similar appended to the file name.
I switched to the Synology system because it's ideal to reconcile since an up-to-date backup set can be shared via SMB. That makes it possible to have an up-to-date OneDrive sync and a mapped drive to an up-to-date backup set on the same machine. After that, it's a matter of comparing two folders as long as care is taken to get a consistent point-in-time for both sets of data.
The only noteworthy thing that I think plays a part is how frequently the tenant reorganizes their data. They're always renaming and moving files and folders to keep things organized. I'd frame it as being frequent, but not unreasonable. The reason I think this is noteworthy is that in cases where I'm able to track the life-cycle of missing files, they seem to "disappear" after being impacted by a directory rename or move operation.
I can't engage with support for this particular tenant because the data is supporting documentation for government work. I can't even share examples or screenshots with the file names AFAIK.
I've seen people complaining about similar issues, but the complaints are from years ago [1]. This [2] caught my eye.
> The root cause for the missing data was due to incorrect representation of the changes from the SharePoint API side.
Veeam RnD team performed an investigation and found that sometimes the SharePoint API mechanism of tracking changes did not track the changes inside the Child files or Folders inside the Site`s list.
Does anyone here reconcile their OneDrive backups well enough to say you're confident you can reliably restore your data with 100% consistency?
Is it possible there's a change tracking bug lurking in the SharePoint API? I don't know anything about how it works, so any insight would be useful. For example, would the OneDrive client and backup clients use the same change tracking? That seems unlikely based on what I'm seeing, but, again, I have no idea how it actually works.
1. https://forums.veeam.com/veeam-backup-for-microsoft-365-f47/community-edition-folders-but-no-files-backed-up-t58862.html https://forums.veeam.com/veeam-backup-for-microsoft-365-f47/...
2. https://forums.veeam.com/veeam-backup-for-microsoft-365-f47/major-issue-with-incomplete-sharepoint-backups-files-missing-and-lost-t78312.html https://forums.veeam.com/veeam-backup-for-microsoft-365-f47/...
- mschuster91 3y ago> I can't engage with support for this particular tenant because the data is supporting documentation for government work. I can't even share examples or screenshots with the file names AFAIK. Your purchase department should have access to their Microsoft-side account representative to get you access to support without breaching legal requirements.
- jnsaff2 3y agoI've been in a few companies which where we had dedicated account managers on MS side. One was even so big that there were MS engineers sitting with our teams. Questions of actual substance were rarely answered satisfactorily, sometimes weeks of chasing got nowhere. With this context I would not hold by breath for them to come back with anything useful. Also having seen the kinds of untested garbage they deploy as services I would not be surprised at all at them silently losing data, I would not even be sure for them ever noticing that they have lost data. For anyone using their storage as backups, I recommend having an inventory (with hashes) and comparing that to whats in MS from time to time. At least you know you are not going crazy, even if MS never trusts your proof about their incompetence.
- jnsaff2 3y agoHell, github has had a lot of outages in the last weeks (webhooks for the most part), I would not be surprised if they'd lose github data .. which I guess would be detected by git eventually but hey, what's gone is gone.
- avannatta 3y agoThat's terrifying. I'm going to spot check our N-able backups more closely
- prmph 3y agoI wouldn't trust any MS cloud product to not lose my data. A year or two ago I lost data backing up my OneDrive files to my local HDD. I was told my backup had succeeded, and so I deleted the files on OneDrive. Later, when I tried to extract some large files from the backup, I saw that they had been replaced with some text files, containing a message that the download of those files failed. What the heck... Was I expected to extract the archive and go through all the files one-by-one (there are thousands of them) to check if every file was properly backed up?