5 ms·

Rsync's defaults are not always enough

38 points by rcarmo 1y ago

1oooqooq 1y ago
rsync is perfect. just have decent fs on both sides.
- rob_c 1y ago
  It's 95-99% of the way there, but most of the corner cases are users wanting it to be sentient on their behalf I'll agree.
baobun 1y ago
Are rsync defaults ever enough? I find "-apHAX" a sane default for most use and memorable enough. (think wardriving) Very common contextuals: -c (as mentioned, when you care about integrity) -n (dryrun) -v (verbose) -z (compress when remoting) Where it applies I usually do the actual syncing without -c and then follow up with -cnv to see if anything is off.
- pabs3 1y ago
  -A implies -p btw
  - imtringued 1y ago
    ap is an acronym for access point
- pabs3 1y ago
  Also, you are missing -S to preserve sparseness btw.
wodenokoto 1y ago
Author doesn’t explain what happened or why the proposed flags will solve the problem.
- stevekemp 1y ago
  Author explained that by default if two files are the same size, and have the same modification date/time, that rsync will assume they're identical, WITHOUT CHECKING THAT. Author clarifies there are flags to change that behaviour, to make it actually compare file contents, and then shares those names. It seems like you didn't read the article.
  - XorNot 1y ago
    They also have to have the same name though. The actual chances of this situation happening and persisting long enough to matter are pretty damn small.
  - rob_c 1y ago
    Genuinely that's a feature not a bug. If you didn't rt"friendly"m the problem explicitly exists between keyboard/vr-headset and chair/standing-desk. This should never be a surprise to people unless this is their first time using Unix.
  - wodenokoto 1y ago
    Short of checking every single byte against each other you need to do some sort of short hand. “Assume these two files are the same, check if they either system is saying they have modified it or check if the size has changed and call it a day” is pretty fair of an assumption and something even I knew RSync is doing and I’ve only used it once in a project 10 years ago. I am sure Rachel also knows this. So, what is the problem? Is data not being synced? Is data being synced too often? And why do these assumptions lead to either happening? What horrors is the author expecting the reader to see when running the suggested command? That is what is not explained in the article.
- mustache_kimono 1y ago
  > Author doesn’t explain what happened or why the proposed flags will solve the problem. Probably because she/he doesn't know. Could be lots of things, because FYI mtime can be modified by the user. Go `touch` a file. In all likelihood, it happens because of a package installation, where a package install sets the same mtime, on a file which has the same sized, but has different file contents. That's where I usually see it. `httm` allows one to dedup snapshot versions by size, then hash the contents of identical sized versions for this very reason. --dedup-by[=<DEDUP_BY>] comparing file versions solely on the basis of size and modify time (the default "metadata" behavior) may return what appear to be "false positives". This is because metadata, specifically modify time and size, is not a precise measure of whether a file has actually changed. A program might overwrite a file with the same contents, and/or a user can simply update the modify time via 'touch'. When specified with the "contents" option, httm compares the actual file contents of same-sized file versions, overriding the default "metadata" only behavior...
reisse 1y ago
Checksumming if not run on a spinning rust should've been default for years now.
throw93849494 1y ago
Rsync also does not work with mounted filesystems, docker, snaps...
- account42 1y ago
  What is that supposed to mean? It absolutely can work both restricted to a single file system or across file systems.
dedicate 1y ago
Man, reading this just makes me wonder again why -c (checksum) isn't the default in rsync by now, especially with SSDs everywhere. Is it really just about that tiny bit of speed?
- 0rzech 1y ago
  Checksums are slow on HDDs and using SSDs for backups is a bad idea, particularly for cold backups, due to data longevity and the tendency for SSDs to experience sudden total failures instead of gradual degradation.
  - Aachen 1y ago
    The -c makes it do more reads, though, not unnecessary writes Degradation won't get worse except when a file changed without metadata having been modified, but then that's exactly what you want
    - 0rzech 1y ago
      Yes, but it's orthogonal to my comment.
- kissgyorgy 1y ago
  It's not a tiny bit at all. For checksum, the whole file need to be read beforehand, the metadata is just a quick lookup. It's order of magnitude slower and eats I/O too.
- account42 1y ago
  Because changing the defaults of command-line tools is generally a shitty thing to do. And the speed difference is only tiny for tiny files.
nithssh 1y ago
There are a lot of reasons why just making a copy of the files you need to another FS is not sufficient as a backup, clearly this is one of those. We need more checks to ensure integrity and robustness. BorgBackup is clearly quite good as an option.
- mustache_kimono 1y ago
  > BorgBackup is clearly quite good as an option. After one enables rsync with checksums, doesn't Borg have the same issue? I believe Borg needs to do the same rolling checksum over all the data, now, as well? ZFS sounds like the better option -- just take the last local snapshot transaction, then compare to the transaction of the last sent snapshot, and send everything in between. And the problem re: Borg and rsync isn't just the cost of reading back and checksumming the data -- for 100,000s of small files (1000s of home directories on spinning rust), it is the speed of those many metadata ops too.
  - fpoling 1y ago
    As with rsync borg does not read files if their timestamp/length do not change since the last backup. And for million files on modern SSD it takes just few seconds to read their metadata.
    - mustache_kimono 1y ago
      > As with rsync borg does not read files if their timestamp/length do not change since the last backup. ...but isn't that the problem described in the article? If that is the case, Borg would seem to the worst of all possible worlds, because now one can't count on its checksums?
      - fpoling 1y ago
        If one worries about bitrot, the backup tools are not good place to detect that. Using a filesystem with native checksums is the way to go. If one worries about silent file modifications that alters content but keep timestamp and length, then this sounds like malware and, as such, the backup tools are not the right tool to deal with that.
kissgyorgy 1y ago
Maybe NixOS spoiled me, but I would not copy system files at all. The only data worth backing up is app state and your files usually under /home or /var/lib/whatever.
- porridgeraisin 1y ago
  Yeah. But the system files add so little to the backup size that I just backup everything except tmpfs and xdg cache anyways. I also reduce backup size by simply not backing up ~/{Pictures,Videos}. I manually put those in the google photos account my phone backs up to since they very rarely change.
- password4321 1y ago
  What are today's best options for programmatically determining the difference between user and system files for normal Windows users?
  - teekert 1y ago
    Everything is system files except for /home and var/lib/whatever in WSL ;) ;) Or, since installing is such a pain, perhaps better consider everything user files ;) ;)
- 1oooqooq 1y ago
  [flagged]
  - Aachen 1y ago
    TFA
    - 1oooqooq 1y ago
      you might want to to read it.
  - wrasee 1y ago
    The article you probably didn’t read.
KWxIUElW8Xt0tD9 1y ago
I moved to ZFS to get some guarantees regarding long-term storage of important data -- e.g. family pictures.
dezgeg 1y ago
I wish checksumming filesystems had some interface to expose the internal checksums. Maybe it wouldn't be useful for rsync though as filesystems should have the freedom to pick the best algorithm (so filesystem checksum of a file on different machines would be allowed to differ e.g. if filesystem block size was different). But so that e.g. git and build systems could use it to tell 'these 2 files under a same directory tree are definitely identical'.
- mustache_kimono 1y ago
  I actually once suggested this to ZFS, and saw some push back! See: https://github.com/issues/created?issue=openzfs%7Czfs%7C14536 https://github.com/issues/created?issue=openzfs%7Czfs%7C1453... Maybe someone else will have better luck than me.
  - hello_computer 1y ago
    If they expose it, that ties them to a particular hash algo. Hash algos are as much art as science, and the opportunities for hardware acceleration vary from chip to chip, so maintaining the leeway to move from one algo to another is kind of important.
    - ThatPlayer 1y ago
      Also similarly is filesystem's transparent compression. The checksum should be on compressed blocks, to check everything without having to decompress them during a scrub. You'd need to match compression settings on both ends. A different number of threads used will change the result too, would probably change depending on the hardware. Would also apply to encryption. Probably shouldn't be using the same encryption key on different filesystems. Or if you're using bcachefs with background compression, compression might not even happen till later.
    - deleted 1y ago
      [deleted]
  - rob_c 1y ago
nottorp 1y ago
> That's totally a problem that you have to worry about, especially if you're using SSDs to hold your bits and those SSDs aren't always being powered. This deserves being in all caps.
dspillett 1y ago
No tool's defaults are always enough. Otherwise they'd be fixed settings not changeable defaults. > "ooh, bit rot" and other things where one of the files has actually become corrupted while "at rest" for whatever reason. Those observers are right! Yep. This is why you verify your backups occasionally. And perhaps your local “static” resources too depending on your accident/attack threat models and “time to recovery” requirements in the event of something going wrong. > the first time you do a forced-checksum run, --dry-run will let you see the changes before it blows anything away, so you can make the call as to which version is the right one! That reads like someone is not doing snapshot backups, or at least not doing backups in a way that means the past snapshots are not protected from being molested by the systems being backed up. This is a mistake, and not one rsync or any other tool can reliably save you from. But yes, --dry-run is a good idea before any config change. Or just generally in combination with a checksum based run as part of a sanity check procedure (though as rsync is my backup tool, I prefer to verify my backups with a different method, checksums generated by another tool, or direct comparison between snapshots/current by another tool, just-in-case a bug in rsync is causing an issue that verification using rsync cannot detect because the bug affects said verification in the same way).
bo0tzz 1y ago
This is one of many reasons to avoid rsync for "backups" and instead use an actual backup tool.

38 points by rcarmo 1y ago

1oooqooq 1y ago

rsync is perfect. just have decent fs on both sides.

rob_c 1y ago
It's 95-99% of the way there, but most of the corner cases are users wanting it to be sentient on their behalf I'll agree.

baobun 1y ago

Are rsync defaults ever enough? I find "-apHAX" a sane default for most use and memorable enough. (think wardriving) Very common contextuals: -c (as mentioned, when you care about integrity) -n (dryrun) -v (verbose) -z (compress when remoting) Where it applies I usually do the actual syncing without -c and then follow up with -cnv to see if anything is off.

pabs3 1y ago
-A implies -p btw
- imtringued 1y ago
  ap is an acronym for access point
pabs3 1y ago
Also, you are missing -S to preserve sparseness btw.

wodenokoto 1y ago

Author doesn’t explain what happened or why the proposed flags will solve the problem.

stevekemp 1y ago
Author explained that by default if two files are the same size, and have the same modification date/time, that rsync will assume they're identical, WITHOUT CHECKING THAT. Author clarifies there are flags to change that behaviour, to make it actually compare file contents, and then shares those names. It seems like you didn't read the article.
- XorNot 1y ago
  They also have to have the same name though. The actual chances of this situation happening and persisting long enough to matter are pretty damn small.
- rob_c 1y ago
  Genuinely that's a feature not a bug. If you didn't rt"friendly"m the problem explicitly exists between keyboard/vr-headset and chair/standing-desk. This should never be a surprise to people unless this is their first time using Unix.
- wodenokoto 1y ago
  Short of checking every single byte against each other you need to do some sort of short hand. “Assume these two files are the same, check if they either system is saying they have modified it or check if the size has changed and call it a day” is pretty fair of an assumption and something even I knew RSync is doing and I’ve only used it once in a project 10 years ago. I am sure Rachel also knows this. So, what is the problem? Is data not being synced? Is data being synced too often? And why do these assumptions lead to either happening? What horrors is the author expecting the reader to see when running the suggested command? That is what is not explained in the article.
mustache_kimono 1y ago
> Author doesn’t explain what happened or why the proposed flags will solve the problem. Probably because she/he doesn't know. Could be lots of things, because FYI mtime can be modified by the user. Go `touch` a file. In all likelihood, it happens because of a package installation, where a package install sets the same mtime, on a file which has the same sized, but has different file contents. That's where I usually see it. `httm` allows one to dedup snapshot versions by size, then hash the contents of identical sized versions for this very reason. --dedup-by[=<DEDUP_BY>] comparing file versions solely on the basis of size and modify time (the default "metadata" behavior) may return what appear to be "false positives". This is because metadata, specifically modify time and size, is not a precise measure of whether a file has actually changed. A program might overwrite a file with the same contents, and/or a user can simply update the modify time via 'touch'. When specified with the "contents" option, httm compares the actual file contents of same-sized file versions, overriding the default "metadata" only behavior...

reisse 1y ago

Checksumming if not run on a spinning rust should've been default for years now.

throw93849494 1y ago

Rsync also does not work with mounted filesystems, docker, snaps...

account42 1y ago
What is that supposed to mean? It absolutely can work both restricted to a single file system or across file systems.

dedicate 1y ago

Man, reading this just makes me wonder again why -c (checksum) isn't the default in rsync by now, especially with SSDs everywhere. Is it really just about that tiny bit of speed?

0rzech 1y ago
Checksums are slow on HDDs and using SSDs for backups is a bad idea, particularly for cold backups, due to data longevity and the tendency for SSDs to experience sudden total failures instead of gradual degradation.
- Aachen 1y ago
  The -c makes it do more reads, though, not unnecessary writes Degradation won't get worse except when a file changed without metadata having been modified, but then that's exactly what you want
  - 0rzech 1y ago
    Yes, but it's orthogonal to my comment.
kissgyorgy 1y ago
It's not a tiny bit at all. For checksum, the whole file need to be read beforehand, the metadata is just a quick lookup. It's order of magnitude slower and eats I/O too.
account42 1y ago
Because changing the defaults of command-line tools is generally a shitty thing to do. And the speed difference is only tiny for tiny files.

nithssh 1y ago

There are a lot of reasons why just making a copy of the files you need to another FS is not sufficient as a backup, clearly this is one of those. We need more checks to ensure integrity and robustness. BorgBackup is clearly quite good as an option.

mustache_kimono 1y ago
> BorgBackup is clearly quite good as an option. After one enables rsync with checksums, doesn't Borg have the same issue? I believe Borg needs to do the same rolling checksum over all the data, now, as well? ZFS sounds like the better option -- just take the last local snapshot transaction, then compare to the transaction of the last sent snapshot, and send everything in between. And the problem re: Borg and rsync isn't just the cost of reading back and checksumming the data -- for 100,000s of small files (1000s of home directories on spinning rust), it is the speed of those many metadata ops too.
- fpoling 1y ago
  As with rsync borg does not read files if their timestamp/length do not change since the last backup. And for million files on modern SSD it takes just few seconds to read their metadata.
  - mustache_kimono 1y ago
    > As with rsync borg does not read files if their timestamp/length do not change since the last backup. ...but isn't that the problem described in the article? If that is the case, Borg would seem to the worst of all possible worlds, because now one can't count on its checksums?
    - fpoling 1y ago
      If one worries about bitrot, the backup tools are not good place to detect that. Using a filesystem with native checksums is the way to go. If one worries about silent file modifications that alters content but keep timestamp and length, then this sounds like malware and, as such, the backup tools are not the right tool to deal with that.

kissgyorgy 1y ago

Maybe NixOS spoiled me, but I would not copy system files at all. The only data worth backing up is app state and your files usually under /home or /var/lib/whatever.

porridgeraisin 1y ago
Yeah. But the system files add so little to the backup size that I just backup everything except tmpfs and xdg cache anyways. I also reduce backup size by simply not backing up ~/{Pictures,Videos}. I manually put those in the google photos account my phone backs up to since they very rarely change.
password4321 1y ago
What are today's best options for programmatically determining the difference between user and system files for normal Windows users?
- teekert 1y ago
  Everything is system files except for /home and var/lib/whatever in WSL ;) ;) Or, since installing is such a pain, perhaps better consider everything user files ;) ;)
1oooqooq 1y ago
[flagged]
- Aachen 1y ago
  TFA
  - 1oooqooq 1y ago
    you might want to to read it.
- wrasee 1y ago
  The article you probably didn’t read.

KWxIUElW8Xt0tD9 1y ago

I moved to ZFS to get some guarantees regarding long-term storage of important data -- e.g. family pictures.

dezgeg 1y ago

I wish checksumming filesystems had some interface to expose the internal checksums. Maybe it wouldn't be useful for rsync though as filesystems should have the freedom to pick the best algorithm (so filesystem checksum of a file on different machines would be allowed to differ e.g. if filesystem block size was different). But so that e.g. git and build systems could use it to tell 'these 2 files under a same directory tree are definitely identical'.

mustache_kimono 1y ago
I actually once suggested this to ZFS, and saw some push back! See: https://github.com/issues/created?issue=openzfs%7Czfs%7C14536 https://github.com/issues/created?issue=openzfs%7Czfs%7C1453... Maybe someone else will have better luck than me.
- hello_computer 1y ago
  If they expose it, that ties them to a particular hash algo. Hash algos are as much art as science, and the opportunities for hardware acceleration vary from chip to chip, so maintaining the leeway to move from one algo to another is kind of important.
  - ThatPlayer 1y ago
    Also similarly is filesystem's transparent compression. The checksum should be on compressed blocks, to check everything without having to decompress them during a scrub. You'd need to match compression settings on both ends. A different number of threads used will change the result too, would probably change depending on the hardware. Would also apply to encryption. Probably shouldn't be using the same encryption key on different filesystems. Or if you're using bcachefs with background compression, compression might not even happen till later.
  - deleted 1y ago
    [deleted]
- rob_c 1y ago

nottorp 1y ago

> That's totally a problem that you have to worry about, especially if you're using SSDs to hold your bits and those SSDs aren't always being powered. This deserves being in all caps.

dspillett 1y ago

No tool's defaults are always enough. Otherwise they'd be fixed settings not changeable defaults. > "ooh, bit rot" and other things where one of the files has actually become corrupted while "at rest" for whatever reason. Those observers are right! Yep. This is why you verify your backups occasionally. And perhaps your local “static” resources too depending on your accident/attack threat models and “time to recovery” requirements in the event of something going wrong. > the first time you do a forced-checksum run, --dry-run will let you see the changes before it blows anything away, so you can make the call as to which version is the right one! That reads like someone is not doing snapshot backups, or at least not doing backups in a way that means the past snapshots are not protected from being molested by the systems being backed up. This is a mistake, and not one rsync or any other tool can reliably save you from. But yes, --dry-run is a good idea before any config change. Or just generally in combination with a checksum based run as part of a sanity check procedure (though as rsync is my backup tool, I prefer to verify my backups with a different method, checksums generated by another tool, or direct comparison between snapshots/current by another tool, just-in-case a bug in rsync is causing an issue that verification using rsync cannot detect because the bug affects said verification in the same way).

bo0tzz 1y ago

This is one of many reasons to avoid rsync for "backups" and instead use an actual backup tool.