4 ms·

Are you sure you understand how VEEAM MS365 actually backs up data? They designed it to look like a differential backup model, but it's not. In fact the only b

Are you sure you understand how VEEAM MS365 actually backs up data? They designed it to look like a differential backup model, but it's not. In fact the only backup you can trust as to its integrity is the first full one. https://helpcenter.veeam.com/docs/vbo365/guide/retention_policy.html https://helpcenter.veeam.com/docs/vbo365/guide/retention_pol... “When a retention policy is applied in backup repositories with the Snapshot-Based Retention type, Veeam Backup for Microsoft 365 removes versions of an item, but not an item itself. Data removal from backup occurs every time the restore point of an item's version in a backup file goes beyond the retention coverage. Eventually, if no more changes were made to an item, Veeam Backup for Microsoft 365 will remove all versions of an item except the latest one. The latest versions and items that were never changed stay in a backup repository with the Snapshot-Based Retention type forever.”

ryan87 3y ago
> They designed it to look like a differential backup model > Eventually, if no more changes were made to an item, Veeam Backup for Microsoft 365 will remove all versions of an item except the latest one. I added the emphasis. I read that to mean I can expect the current point-in-time (aka right now) to be identical to my live data. I'm only trying to reconcile the most recent set of data, so that retention policy shouldn't make any difference, right?
- doctorpangloss 3y ago
  > Veeam As my 2 year old would say, "uh oh." Are you saying your backup solution is: - there's a folder somewhere whose contents are "synchronized", crying laugh emojis galore, using a "Dogshit" protocol, as it appears in the programs Microsoft OneShit or DropShit or Shit.net or Google Shit. - you have a machine that has a "copy" of a dogshit-synchronized file system, perhaps using a vendor's implementation of Dogshit, such as Synology Dogshit Manager, a piece of software with which I am intimately acquainted - that machine "regularly" "backs up" its "copy" via a procedure known confusingly as "dogShit" - wait a minute, the files aren't right! I hear you. This is very surprising, but it would occur no matter which file sharing system you use, so long as it's based on Dogshit and/or dogShit. What is the specific flaw with dogShit? The simplest explanation for everything you observe, like the constant moving by multiple users, is that while using ordinary file system enumerations of the form "walk," a folder may be moved in a way such that it is never visited by "walk," even though the destination of the moved folder is still within the root directory of the walk. You can easily verify this in Python. Okay, so now that I solved your bug in dogShit, what should you use instead? Snapshotters have already solved this problem, you have to create a dedicated filesystem for the shared directory on your Synology, then snapshot the whole filesystem, which will only have issues with "missing" since the moment it started but never traversing the filesystem incorrectly. Then you can backup the file system snapshots to S3. In Synology you can achieve this with btrfs and some elbow grease. But the right answer is to not use dogshit. At the end of the day the people who are authoring Dogshit file sharing products, they should be giving you a backup approach, not demanding you do it ad-hoc.
pengaru 3y ago
> They designed it to look like a differential backup model, but it's not. The quote you pasted makes it sound like a reverse-differential incremental backup, like rdiff-backup implements (poorly). The most recent backup is a snapshot of the tree, restore points having changes are diffs applied cumulatively in reverse chronological order atop the current snapshot. You can simply toss out restore points starting from the tail older than the retention policy. How is that not a differential backup model?
- still-old 3y ago
  In case anyone comes back to this, it's because any snapshot who's retention date is out of the retention period can get removed. When the original item is out of the retention period it can also get removed. There is no cumulation of diffs, its just diffs from the original object. You might notice that the quoted text of my original post has also been changed on the link.

ryan87 3y ago

> They designed it to look like a differential backup model > Eventually, if no more changes were made to an item, Veeam Backup for Microsoft 365 will remove all versions of an item except the latest one. I added the emphasis. I read that to mean I can expect the current point-in-time (aka right now) to be identical to my live data. I'm only trying to reconcile the most recent set of data, so that retention policy shouldn't make any difference, right?

doctorpangloss 3y ago
> Veeam As my 2 year old would say, "uh oh." Are you saying your backup solution is: - there's a folder somewhere whose contents are "synchronized", crying laugh emojis galore, using a "Dogshit" protocol, as it appears in the programs Microsoft OneShit or DropShit or Shit.net or Google Shit. - you have a machine that has a "copy" of a dogshit-synchronized file system, perhaps using a vendor's implementation of Dogshit, such as Synology Dogshit Manager, a piece of software with which I am intimately acquainted - that machine "regularly" "backs up" its "copy" via a procedure known confusingly as "dogShit" - wait a minute, the files aren't right! I hear you. This is very surprising, but it would occur no matter which file sharing system you use, so long as it's based on Dogshit and/or dogShit. What is the specific flaw with dogShit? The simplest explanation for everything you observe, like the constant moving by multiple users, is that while using ordinary file system enumerations of the form "walk," a folder may be moved in a way such that it is never visited by "walk," even though the destination of the moved folder is still within the root directory of the walk. You can easily verify this in Python. Okay, so now that I solved your bug in dogShit, what should you use instead? Snapshotters have already solved this problem, you have to create a dedicated filesystem for the shared directory on your Synology, then snapshot the whole filesystem, which will only have issues with "missing" since the moment it started but never traversing the filesystem incorrectly. Then you can backup the file system snapshots to S3. In Synology you can achieve this with btrfs and some elbow grease. But the right answer is to not use dogshit. At the end of the day the people who are authoring Dogshit file sharing products, they should be giving you a backup approach, not demanding you do it ad-hoc.

pengaru 3y ago

> They designed it to look like a differential backup model, but it's not. The quote you pasted makes it sound like a reverse-differential incremental backup, like rdiff-backup implements (poorly). The most recent backup is a snapshot of the tree, restore points having changes are diffs applied cumulatively in reverse chronological order atop the current snapshot. You can simply toss out restore points starting from the tail older than the retention policy. How is that not a differential backup model?

still-old 3y ago
In case anyone comes back to this, it's because any snapshot who's retention date is out of the retention period can get removed. When the original item is out of the retention period it can also get removed. There is no cumulation of diffs, its just diffs from the original object. You might notice that the quoted text of my original post has also been changed on the link.