9 ms·

Correct Backups Require Filesystem Snapshots

50 points by cyounkins 4y ago

ReactiveJelly 4y ago
Completely true But also, if you have no backups, incorrect ragged backups of only non-changing files are a _hell_ of a lot better than no backups at all. And if you get in the habit of doing incremental backups every day, you might be lucky enough to find a non-corrupted version of a file that's a few days stale.
- drewzero1 4y ago
  For a long time I didn't know where to start with backups, because I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) Eventually I realized that some backups are better than no backups at all, and prioritized getting _something_ in place to at the very least have copies of important files copied regularly to another drive somewhere.
  - michaelt 4y ago
    > I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) It depends a lot on your use case. If you're backing up a production database system that's in constant use, then the files are almost certain to change while the backup is in progress. And a backup of everything on the database server except the data probably isn't what you want! On the other hand, if I'm backing up my personal computer, my family photos aren't getting regularly overwritten - and if the backup of my web browser cache is inconsistent? So be it.
    - aidenn0 4y ago
      It's been a while, but with e.g. Oracle 10, it was specifically stated in the docs that backing up the data-file(s), followed by the archive-logs, you would end up with a recoverable backup. I'm not sure if other databases are designed that way.
      - professoretc 4y ago
        I've done it with SQL Server, but it wasn't automatic. I had to manually repair or remove the DB pages that were in use when the backup was made.
        aidenn0 4y ago
        I think with Oracle you needed to replay the archive log; not sure if that was via rman or just a mount option. Either way it was identical to crash recovery
    - drewzero1 4y ago
      In my case it was mostly office PCs that might have important documents or spreadsheets on them. I've been working on a redundant, networked backup solution to serve the whole office, but realized that I was building a henhouse while the chickens were already running loose. So I ordered some USB hard drives just to make sure things were getting copied. Maintaining a super basic (but regularly verified) hourly incremental backup system has spared the people who use those PCs a lot of grief, plus saved me time trying to recover lost data.
liuliu 4y ago
Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a file system that supports transaction, requires every application to use a competent database underneath (SQLite) at least. And even with transactional support in a file system or using a database, you need every application to have the correct business logic that does the transaction correctly as well. All-in-all, correct backup cannot be solved universally without knowing all the applications. The best we can do is to probabilistically avoid obvious issues, i.e. using FS-level snapshot.
- cyounkins 4y ago
  Yes, you are correct in that it does not guarantee correctness. Point-in-time snapshots are necessary but not sufficient. Without them, the possible corruption scenarios are infinite and cannot be handled or even detected. With them, it is up to the applications to do the right thing in what is equivalent to a power loss event.
- rzzzt 4y ago
  The Volume Shadow Copy Service mentioned at the end has support for application-level notifications via writers, the most prominent one being SQL Writer: https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/sql-server-vss-writer-backup-guide?view=sql-server-ver15#sql-writer https://docs.microsoft.com/en-us/sql/relational-databases/ba...
- adrianmonk 4y ago
  One way of looking at it is that applications already need to be able to survive power loss, a crash of the computer, a crash of the application, or a forcible kill of the application. If they can survive those, they can probably survive a filesystem snapshot being taken at just the wrong moment.
  - 4y ago
ozim 4y ago
Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at the time. When my laptop dies I probably can just remember what was needed to be done and redo the work. Restoring file system snapshot would usually be much more hassle than filling in single invoice again or downloading it from some provider again. For web applications there is usually SLA where they specify how much data can be lost like 1 hour or 30 mins - but no one will realistically guarantee "no data lost" - because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway.
- cyounkins 4y ago
  Sure, I've been accused of being overly concerned with correctness. :-) > because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway. On copy-on-write filesystems, snapshots are nearly instantaneous. > time than setting up and keeping file system snapshots running As shown in the code snippet, pretty much all it takes is a few snapshot commands around the backup command and changing the source directory to the snapshot mount point.
  - ozim 4y ago
    Different people care about different things :) I also think about snapshots differently. For me snapshot is actual copy stored on a different hard drive or other medium. So copy operation is never going to be instant. Scenario in the article is power loss so that is also not something I worry about that much. Mostly I worry about hardware failure where I would not be able to turn on my laptop/server again. For power loss UPS or battery in laptop is my go to solution.
    - dsr_ 4y ago
      The snapshot takes a second to generate; then you can take as long as your hardware needs to get it somewhere else.
mst 4y ago
Given ZFS it's potentially worth looking at https://github.com/jimsalterjrs/sanoid https://github.com/jimsalterjrs/sanoid
fudgy 4y ago
https://www.arqbackup.com/ https://www.arqbackup.com/ uses snapshots both on macOS and Windows!
throw7 4y ago
Wouldn't you need to quiesce the db or application before the snapshot?
- blibble 4y ago
  yep, otherwise you risk some state being in memory and not on the disk
  - SoftTalker 4y ago
    Exactly. A snapshot is a point in time on the storage media. It allows you to make a backup as if the backup occurred at that point in time, i.e. files are not changing while the backup is running. A snapshot is not a guarantee that any particular file is consistent with the state of the application that is using it. It's still a backup of an "open" file with all the potential issues that implies.
  - topspin 4y ago
    Your answer is too definitive. Applications and databases can indeed suffer power loss while losing nothing of value. An ACID database used by a correctly designed application will lose nothing on power loss; state "in memory" is uncommitted in such a system effectively doesn't exist. It never happened and, furthermore, no one cares. The term of art is "crash consistent," and any ACID database must preserve all committed state across events such as power loss. Such a database is correctly backed up when copying a simultaneous point in time snapshot across all involved volumes. Not all databases are truly ACID. Lots of software relies on uncommitted database state. But we're talking about a solved problem here; if you require ACID behavior the means to achieve exactly that are available. Any exception to that statement, including hardware misfeatures or lack of two phase commit across databases, is equivalent to "incorrectly designed." In a correctly designed system quiescing the database isn't necessary, but might still be used as a precaution or a performance optimization.
    - blibble 4y ago
      this is fantastic and all for carefully written databases (none of which have ever had any bugs) unfortuntely most applications aren't like that and assume the filesystem is both atomic and durable
nix0n 4y ago
The other option is downtime: boot into another OS, so you can mount the disk to be backed up as read-only.
aborsy 4y ago
Ubuntu supports ZFS with one click at installation time. You can snapshot your system every 15 minutes or as you desire.
- ComputerGuru 4y ago
  Note that it’s not really a supported option and was supposed to be removed from the installer for the latest release but that was cancelled because there wasn’t enough time to do so before the freeze. It’s certain to be dropped in the next release. So by all means, play with it but don’t consider it to be stable. https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1966773 https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1966... https://code.launchpad.net/~ubuntu-installer/ubiquity/+git/ubiquity/+ref/remove_zsys https://code.launchpad.net/~ubuntu-installer/ubiquity/+git/u... Edit: fixed link.
notacoward 4y ago
Should clarify that OP is talking about a consumer use case. In the enterprise space, quiescing applications and databases during a snapshot was standard at least as far back as 1992 when I started working on HA software which would (among its other functions) automate the process. For quite a few years after that, I worked on systems where block-level snapshots were more common (and performed better) than filesystem-level. Filesystem snapshots are great, and it's also great to warn people about the dangers of doing backups without some sort of protection against in-flight changes, but the title as written isn't quite correct.
- viraptor 4y ago
  For quite a few years after 1992 we didn't really have simple solutions that could do atomic snapshots. I'm not sure about specialised enterprise cases, but for Linux consumers, the first one would be lvm right? (And that was not write-barrier-safe until 2010)
compsciphd 4y ago
even with file system snapshots, your backup can be corrupt. the example of the browser profile is a classic case. imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt". I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction. without such a mechanism, you are always going to be at risk with snapshots. In https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf we avoided this problem by combining 2 mechanisms without having modifying applications with such a service 1) we used a log structured file system (that was inherently a snapshot, every log entry was individually mountable) and 2) we used a checkpoint/restart mechanism that saved process state and enabled us to restart the processes combined with the file system state as it was at checkpoint time. (Checkpoint would also sync all dirty pages to disk and that fs state after the sync is what we tied to the checkpoint state). So when a process would be resumed, the file system would look exactly as the process expected it, even if the process was in the middle of what can be referred to a transaction. But that only worked because the processes were restored along with file system, if we only restored the file system, it could be inconsistent.
- AshamedCaptain 4y ago
  The worst part is that the article itself says this. This is exactly like "pulling the power on the running system" and then imaging the hard disk. Even if it happens to work, that is not really a good idea.
- grdomzal 4y ago
  > I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction. Former maintainer of VSS here. Yes, that's exactly right. In fact, filesystem snapshots are usually not enough for true application level consistency. As others have noted elsewhere, a filesystem snapshot is the equivalent to yanking the power cord out of the back of your computer. It's good, assuming your filesystem does atomic writes / copy-on-write / write-to-new. But we can do even better. Imagine I have two databases - one traditional relational DB storing my app content, and a second log database. You want to keep these in sync. Well, good luck doing this with the filesystem alone. Usually your DBMS will need to be involved as well, and this is where the VSS "writer" concept comes in. When a snapshot is being taken, applications such as SQL will be invited to participate. Typically, this means they'll start to hold up writes so that things will be quieter for the snapshot. But they will also have a chance, after the fact, and to actually clean up the snapshot itself. In this case, the DBMS could roll back the log database to then match the content database. It's correct that NTFS doesn't support snapshots natively, but Windows has the volsnap.sys driver that takes care of it. For all intents and purposes, NTFS does support copy-on-write snapshots. Complicated? Sure. But it was quite capable, and actually a pretty cool (but sadly under appreciated) piece of technology.
Freaky 4y ago
I wrote https://github.com/Freaky/zfsnapr https://github.com/Freaky/zfsnapr a few months ago so I could finally have point-in-time consistent Borg backups with ZFS snapshots, without having the mess of teaching Borg where every .zfs directory was. It recursively snapshots mounted pools, and recursively mounts snapshots of the mounted datasets into a target ready to point your backup tools at. I do so via a chroot so I didn't need to make any changes to my Borg setup - just to how I run it.
dboreham 4y ago
Note the article seems to be considering desktop/laptop systems. If you're backing up a server running a database you have another whole set of problems, but generally you won't need PIT snapshot semantics from the filesystem. You get that from the database's storage engine (e.g. via write ahead logging). Backup in that context is done in cahoots with the storage engine such that a consistent-after-recovery set of filesystem data is backed up. Databases such as Oracle, SQLServer and PostgreSQL existed long before filesystems with consistent snapshot capability were common.
bretpiatt 4y ago
A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode). A volume snapshot if you're technical enough to split up OS and application data volumes on your local drive can potentially help but you still have registry or other issues. The best way to handle backups, from the perspective of a backup company CEO, is to backup your critical data, and that could include application configurations. Browsers support exporting your profile, good backup software let's you run a pre-backup job script and post-job script, so you can export your browser profile, lock certain files or folders during a backup, etc. to get a good recovery point while minimizing risk of data corruption. Way longer topic than a comment here. If you do want full system state, pick a hypervisor (Parallels, etc.) and then backup the full VM each time you power the VM down. It'll make for much larger backups so restores are slower and storing all the recovery points will cost you a bit more too.
- jiggawatts 4y ago
  This is 100% false and nobody should ever follow this advice. I use snapshots all the time because Windows uses them automatically to get complete and consistent backups. Snapshots are always better then no snapshots. Full backups are always safer than partial backups. It takes just ONE forgotten path or file to make a backup completely impossible to restore. Don’t be a smart ass with backups. Just don’t.
  - bretpiatt 4y ago
    A full system snapshot is not 100% always the correct solution, data protection is complicated. An example hypothetical, you have deeply embedded malware and you're unsure when it infected the system, it could have been sitting latent for months. In this situation you have to restore back and lose months of application data or restore a recent snapshot and then try to remove the malware in a clean room. It is much safer and easier to start from a clean OS and then restoring only the data on the system which is much simpler to scan for infection and sanitize.
- iggldiggl 4y ago
Havoc 4y ago
>On Linux I strongly recommend ZFS due to its long history of reliability I've not been seeing any of this much vaunted reliability. 3 out of 3 of the last drives I lost data on were ZFS. Meanwhile NTFS and EXT4 has been fairly trouble free. Though to be fair hard to tell how much of that is TrueNAS vs ZFS.
- XorNot 4y ago
  This seems very unlikely without a lot of contributing factors. Unless you weren't running any RAID in which case ZFS didn't lose you data but it would've been quite upfront about corruption when it happened.
  - Havoc 4y ago
    >This seems very unlikely That's what I thought given that it does have a good rep. Haven't given up on it yet, but definitely less trusting - 3 different drives in 3 different ways wasn't expected. That said - all consumer class gear in janky environments so there may very well be other factors lurking.
    - ek750 4y ago
      What was the layout of the ZFS pools? I've been using RAIDZ1 and RAIDZ2 for years now on ubuntu, and luckily have not detected any errors on my monthly scrubs. I did have bitrot on older drives previously when I just relied on EXT4, only to find out when trying to access an old file. I'm using a Supermicro board, with ECC RAM, but anywhere from 4 to 6 drives in the pool (upgraded over 6 or 7 years now).
      - kaba0 4y ago
        To my knowledge ZFS indeed has a very very good track record, but Ubuntu did have an incorrect patch for it at a time, as I was enlightened by this comment in the thread: https://news.ycombinator.com/item?id=31292568 https://news.ycombinator.com/item?id=31292568 Perhaps ubuntu was the culprit, not ZFS?
NewEntryHN 4y ago
Nobody cares about correct backups. Backup your pictures and payslips and whatever paper you're working on.

50 points by cyounkins 4y ago

Completely true But also, if you have no backups, incorrect ragged backups of only non-changing files are a _hell_ of a lot better than no backups at all. And if you get in the habit of doing incremental backups every day, you might be lucky enough to find a non-corrupted version of a file that's a few days stale.

drewzero1 4y ago
For a long time I didn't know where to start with backups, because I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) Eventually I realized that some backups are better than no backups at all, and prioritized getting _something_ in place to at the very least have copies of important files copied regularly to another drive somewhere.
- michaelt 4y ago
  > I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) It depends a lot on your use case. If you're backing up a production database system that's in constant use, then the files are almost certain to change while the backup is in progress. And a backup of everything on the database server except the data probably isn't what you want! On the other hand, if I'm backing up my personal computer, my family photos aren't getting regularly overwritten - and if the backup of my web browser cache is inconsistent? So be it.
  - aidenn0 4y ago
    It's been a while, but with e.g. Oracle 10, it was specifically stated in the docs that backing up the data-file(s), followed by the archive-logs, you would end up with a recoverable backup. I'm not sure if other databases are designed that way.
    - professoretc 4y ago
      I've done it with SQL Server, but it wasn't automatic. I had to manually repair or remove the DB pages that were in use when the backup was made.
      - aidenn0 4y ago
        I think with Oracle you needed to replay the archive log; not sure if that was via rman or just a mount option. Either way it was identical to crash recovery
  - drewzero1 4y ago
    In my case it was mostly office PCs that might have important documents or spreadsheets on them. I've been working on a redundant, networked backup solution to serve the whole office, but realized that I was building a henhouse while the chickens were already running loose. So I ordered some USB hard drives just to make sure things were getting copied. Maintaining a super basic (but regularly verified) hourly incremental backup system has spared the people who use those PCs a lot of grief, plus saved me time trying to recover lost data.

liuliu 4y ago

Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a file system that supports transaction, requires every application to use a competent database underneath (SQLite) at least. And even with transactional support in a file system or using a database, you need every application to have the correct business logic that does the transaction correctly as well. All-in-all, correct backup cannot be solved universally without knowing all the applications. The best we can do is to probabilistically avoid obvious issues, i.e. using FS-level snapshot.

cyounkins 4y ago
Yes, you are correct in that it does not guarantee correctness. Point-in-time snapshots are necessary but not sufficient. Without them, the possible corruption scenarios are infinite and cannot be handled or even detected. With them, it is up to the applications to do the right thing in what is equivalent to a power loss event.
rzzzt 4y ago
The Volume Shadow Copy Service mentioned at the end has support for application-level notifications via writers, the most prominent one being SQL Writer: https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/sql-server-vss-writer-backup-guide?view=sql-server-ver15#sql-writer https://docs.microsoft.com/en-us/sql/relational-databases/ba...
adrianmonk 4y ago
One way of looking at it is that applications already need to be able to survive power loss, a crash of the computer, a crash of the application, or a forcible kill of the application. If they can survive those, they can probably survive a filesystem snapshot being taken at just the wrong moment.
- 4y ago

ozim 4y ago

Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at the time. When my laptop dies I probably can just remember what was needed to be done and redo the work. Restoring file system snapshot would usually be much more hassle than filling in single invoice again or downloading it from some provider again. For web applications there is usually SLA where they specify how much data can be lost like 1 hour or 30 mins - but no one will realistically guarantee "no data lost" - because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway.

cyounkins 4y ago
Sure, I've been accused of being overly concerned with correctness. :-) > because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway. On copy-on-write filesystems, snapshots are nearly instantaneous. > time than setting up and keeping file system snapshots running As shown in the code snippet, pretty much all it takes is a few snapshot commands around the backup command and changing the source directory to the snapshot mount point.
- ozim 4y ago
  Different people care about different things :) I also think about snapshots differently. For me snapshot is actual copy stored on a different hard drive or other medium. So copy operation is never going to be instant. Scenario in the article is power loss so that is also not something I worry about that much. Mostly I worry about hardware failure where I would not be able to turn on my laptop/server again. For power loss UPS or battery in laptop is my go to solution.
  - dsr_ 4y ago
    The snapshot takes a second to generate; then you can take as long as your hardware needs to get it somewhere else.

mst 4y ago

Given ZFS it's potentially worth looking at https://github.com/jimsalterjrs/sanoid https://github.com/jimsalterjrs/sanoid

fudgy 4y ago

https://www.arqbackup.com/ https://www.arqbackup.com/ uses snapshots both on macOS and Windows!

throw7 4y ago

Wouldn't you need to quiesce the db or application before the snapshot?

blibble 4y ago
yep, otherwise you risk some state being in memory and not on the disk
- SoftTalker 4y ago
  Exactly. A snapshot is a point in time on the storage media. It allows you to make a backup as if the backup occurred at that point in time, i.e. files are not changing while the backup is running. A snapshot is not a guarantee that any particular file is consistent with the state of the application that is using it. It's still a backup of an "open" file with all the potential issues that implies.
- topspin 4y ago
  Your answer is too definitive. Applications and databases can indeed suffer power loss while losing nothing of value. An ACID database used by a correctly designed application will lose nothing on power loss; state "in memory" is uncommitted in such a system effectively doesn't exist. It never happened and, furthermore, no one cares. The term of art is "crash consistent," and any ACID database must preserve all committed state across events such as power loss. Such a database is correctly backed up when copying a simultaneous point in time snapshot across all involved volumes. Not all databases are truly ACID. Lots of software relies on uncommitted database state. But we're talking about a solved problem here; if you require ACID behavior the means to achieve exactly that are available. Any exception to that statement, including hardware misfeatures or lack of two phase commit across databases, is equivalent to "incorrectly designed." In a correctly designed system quiescing the database isn't necessary, but might still be used as a precaution or a performance optimization.
  - blibble 4y ago
    this is fantastic and all for carefully written databases (none of which have ever had any bugs) unfortuntely most applications aren't like that and assume the filesystem is both atomic and durable

nix0n 4y ago

The other option is downtime: boot into another OS, so you can mount the disk to be backed up as read-only.

aborsy 4y ago

Ubuntu supports ZFS with one click at installation time. You can snapshot your system every 15 minutes or as you desire.

ComputerGuru 4y ago
Note that it’s not really a supported option and was supposed to be removed from the installer for the latest release but that was cancelled because there wasn’t enough time to do so before the freeze. It’s certain to be dropped in the next release. So by all means, play with it but don’t consider it to be stable. https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1966773 https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1966... https://code.launchpad.net/~ubuntu-installer/ubiquity/+git/ubiquity/+ref/remove_zsys https://code.launchpad.net/~ubuntu-installer/ubiquity/+git/u... Edit: fixed link.

notacoward 4y ago

Should clarify that OP is talking about a consumer use case. In the enterprise space, quiescing applications and databases during a snapshot was standard at least as far back as 1992 when I started working on HA software which would (among its other functions) automate the process. For quite a few years after that, I worked on systems where block-level snapshots were more common (and performed better) than filesystem-level. Filesystem snapshots are great, and it's also great to warn people about the dangers of doing backups without some sort of protection against in-flight changes, but the title as written isn't quite correct.

viraptor 4y ago
For quite a few years after 1992 we didn't really have simple solutions that could do atomic snapshots. I'm not sure about specialised enterprise cases, but for Linux consumers, the first one would be lvm right? (And that was not write-barrier-safe until 2010)

compsciphd 4y ago

even with file system snapshots, your backup can be corrupt. the example of the browser profile is a classic case. imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt". I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction. without such a mechanism, you are always going to be at risk with snapshots. In https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf we avoided this problem by combining 2 mechanisms without having modifying applications with such a service 1) we used a log structured file system (that was inherently a snapshot, every log entry was individually mountable) and 2) we used a checkpoint/restart mechanism that saved process state and enabled us to restart the processes combined with the file system state as it was at checkpoint time. (Checkpoint would also sync all dirty pages to disk and that fs state after the sync is what we tied to the checkpoint state). So when a process would be resumed, the file system would look exactly as the process expected it, even if the process was in the middle of what can be referred to a transaction. But that only worked because the processes were restored along with file system, if we only restored the file system, it could be inconsistent.

AshamedCaptain 4y ago
The worst part is that the article itself says this. This is exactly like "pulling the power on the running system" and then imaging the hard disk. Even if it happens to work, that is not really a good idea.
grdomzal 4y ago
> I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction. Former maintainer of VSS here. Yes, that's exactly right. In fact, filesystem snapshots are usually not enough for true application level consistency. As others have noted elsewhere, a filesystem snapshot is the equivalent to yanking the power cord out of the back of your computer. It's good, assuming your filesystem does atomic writes / copy-on-write / write-to-new. But we can do even better. Imagine I have two databases - one traditional relational DB storing my app content, and a second log database. You want to keep these in sync. Well, good luck doing this with the filesystem alone. Usually your DBMS will need to be involved as well, and this is where the VSS "writer" concept comes in. When a snapshot is being taken, applications such as SQL will be invited to participate. Typically, this means they'll start to hold up writes so that things will be quieter for the snapshot. But they will also have a chance, after the fact, and to actually clean up the snapshot itself. In this case, the DBMS could roll back the log database to then match the content database. It's correct that NTFS doesn't support snapshots natively, but Windows has the volsnap.sys driver that takes care of it. For all intents and purposes, NTFS does support copy-on-write snapshots. Complicated? Sure. But it was quite capable, and actually a pretty cool (but sadly under appreciated) piece of technology.

Freaky 4y ago

I wrote https://github.com/Freaky/zfsnapr https://github.com/Freaky/zfsnapr a few months ago so I could finally have point-in-time consistent Borg backups with ZFS snapshots, without having the mess of teaching Borg where every .zfs directory was. It recursively snapshots mounted pools, and recursively mounts snapshots of the mounted datasets into a target ready to point your backup tools at. I do so via a chroot so I didn't need to make any changes to my Borg setup - just to how I run it.

dboreham 4y ago

Note the article seems to be considering desktop/laptop systems. If you're backing up a server running a database you have another whole set of problems, but generally you won't need PIT snapshot semantics from the filesystem. You get that from the database's storage engine (e.g. via write ahead logging). Backup in that context is done in cahoots with the storage engine such that a consistent-after-recovery set of filesystem data is backed up. Databases such as Oracle, SQLServer and PostgreSQL existed long before filesystems with consistent snapshot capability were common.

bretpiatt 4y ago

A system snapshot for a personal computer is potentially a really difficult thing to restore (ex. you break the device and need a new one, unless it matches exactly restoring snapshot likely puts you in OS recovery mode). A volume snapshot if you're technical enough to split up OS and application data volumes on your local drive can potentially help but you still have registry or other issues. The best way to handle backups, from the perspective of a backup company CEO, is to backup your critical data, and that could include application configurations. Browsers support exporting your profile, good backup software let's you run a pre-backup job script and post-job script, so you can export your browser profile, lock certain files or folders during a backup, etc. to get a good recovery point while minimizing risk of data corruption. Way longer topic than a comment here. If you do want full system state, pick a hypervisor (Parallels, etc.) and then backup the full VM each time you power the VM down. It'll make for much larger backups so restores are slower and storing all the recovery points will cost you a bit more too.

jiggawatts 4y ago
This is 100% false and nobody should ever follow this advice. I use snapshots all the time because Windows uses them automatically to get complete and consistent backups. Snapshots are always better then no snapshots. Full backups are always safer than partial backups. It takes just ONE forgotten path or file to make a backup completely impossible to restore. Don’t be a smart ass with backups. Just don’t.
- bretpiatt 4y ago
  A full system snapshot is not 100% always the correct solution, data protection is complicated. An example hypothetical, you have deeply embedded malware and you're unsure when it infected the system, it could have been sitting latent for months. In this situation you have to restore back and lose months of application data or restore a recent snapshot and then try to remove the malware in a clean room. It is much safer and easier to start from a clean OS and then restoring only the data on the system which is much simpler to scan for infection and sanitize.
iggldiggl 4y ago

Havoc 4y ago

>On Linux I strongly recommend ZFS due to its long history of reliability I've not been seeing any of this much vaunted reliability. 3 out of 3 of the last drives I lost data on were ZFS. Meanwhile NTFS and EXT4 has been fairly trouble free. Though to be fair hard to tell how much of that is TrueNAS vs ZFS.

XorNot 4y ago
This seems very unlikely without a lot of contributing factors. Unless you weren't running any RAID in which case ZFS didn't lose you data but it would've been quite upfront about corruption when it happened.
- Havoc 4y ago
  >This seems very unlikely That's what I thought given that it does have a good rep. Haven't given up on it yet, but definitely less trusting - 3 different drives in 3 different ways wasn't expected. That said - all consumer class gear in janky environments so there may very well be other factors lurking.
  - ek750 4y ago
    What was the layout of the ZFS pools? I've been using RAIDZ1 and RAIDZ2 for years now on ubuntu, and luckily have not detected any errors on my monthly scrubs. I did have bitrot on older drives previously when I just relied on EXT4, only to find out when trying to access an old file. I'm using a Supermicro board, with ECC RAM, but anywhere from 4 to 6 drives in the pool (upgraded over 6 or 7 years now).
    - kaba0 4y ago
      To my knowledge ZFS indeed has a very very good track record, but Ubuntu did have an incorrect patch for it at a time, as I was enlightened by this comment in the thread: https://news.ycombinator.com/item?id=31292568 https://news.ycombinator.com/item?id=31292568 Perhaps ubuntu was the culprit, not ZFS?

NewEntryHN 4y ago

Nobody cares about correct backups. Backup your pictures and payslips and whatever paper you're working on.