4 ms·

A major problem with rsync is it relies on filenames matching at the source and destination for an accelerated delta transfer to occur. If a file gets renamed

A major problem with rsync is it relies on filenames matching at the source and destination for an accelerated delta transfer to occur. If a file gets renamed or copied, modified or not, rsync will transfer the whole file. They added --fuzzy to try improve on this situation, but it generally only helps with unmodified files copied/renamed within the same directory. You can build robust solutions around rsync that work efficiently most of the time, but pathological cases such as users having a huge file regularly regenerated using something like a $(date --iso-8601=seconds) filename (think mysqldump backups) are surprisingly common once you have enough users. Even if you can convince them to uncompress such files for delta transfer sake, something as common as a versioned filename prevents rsync from automatically finding the relationship for use in the delta transfer.

catmanjan 6y ago
How else could they do it? All systems I've seen that have relied on files on a file system use the file name as the "primary key" I believe you can use journalling to detect file name change events but I don't think it scales well
- pengaru 6y ago
  Who is "they"? Are you asking about rysnc developers or filesystem developers? For a filesystem the developers are free to innovate on any form of filesystem-wide block checksumming and content-addressed deduplication approach. This kind of global block-level content awareness is entirely decoupled from filenames and paths. Using rsync to underpin a filesystem strikes me as a gross hack done by developers punting on the real challenges of modern filesystem development. WRT rsync doing better than --fuzzy, the options are more limited since it's a userspace tool working within the confines of filesystem APIs like POSIX.
  - catmanjan 6y ago
    So basically there is no good solution at the moment, and a good solution would probably involve enhancing the underlying filesystem - this was the same conclusion I came to
- ComodoHacker 6y ago
  For example, zpaq splits files in chunks and hashes them, so it can deduplicate across files.
BatteryMountain 6y ago
So we need a new tool that is basically git and rsync rolled into one. Might be easier to use something like zfs and use it's features to build a tool that behaves almost like git. Then a (zfs + rsync + [zfs-git hybrid]) monster is born, which may or may not work. Fyi have a look at zfs. It has some neat traits/behaviours. So my thinking is, someone should write a database that uses gazzilions of files (instead of a file per table), a tiny file for each cell/field in a table. Then use zfs for versioning of data (or git). Zfs can already do software mirroring to drive pools on the machine, so maybe add rsync for remote pools. Optimize it for SSD's and you have a filesystem-database-replicating-versioned -based store. Maybe just need a nice cli/gui for zfs to make it easier/safer to expose more of it's interesting features.
- zcw100 6y ago
  “So we need a new tool that is basically git and rsync rolled into one.” That sounds a bit like IPFS
jiggawatts 6y ago
Windows Server has a variant of the Rsync algorithm that it uses for its Distributed File System Replication (DFS-R) that can look for up to 4 similar files. It works okay-ish in practice, and is better than nothing.

catmanjan 6y ago

How else could they do it? All systems I've seen that have relied on files on a file system use the file name as the "primary key" I believe you can use journalling to detect file name change events but I don't think it scales well

pengaru 6y ago
Who is "they"? Are you asking about rysnc developers or filesystem developers? For a filesystem the developers are free to innovate on any form of filesystem-wide block checksumming and content-addressed deduplication approach. This kind of global block-level content awareness is entirely decoupled from filenames and paths. Using rsync to underpin a filesystem strikes me as a gross hack done by developers punting on the real challenges of modern filesystem development. WRT rsync doing better than --fuzzy, the options are more limited since it's a userspace tool working within the confines of filesystem APIs like POSIX.
- catmanjan 6y ago
  So basically there is no good solution at the moment, and a good solution would probably involve enhancing the underlying filesystem - this was the same conclusion I came to
ComodoHacker 6y ago
For example, zpaq splits files in chunks and hashes them, so it can deduplicate across files.

BatteryMountain 6y ago

So we need a new tool that is basically git and rsync rolled into one. Might be easier to use something like zfs and use it's features to build a tool that behaves almost like git. Then a (zfs + rsync + [zfs-git hybrid]) monster is born, which may or may not work. Fyi have a look at zfs. It has some neat traits/behaviours. So my thinking is, someone should write a database that uses gazzilions of files (instead of a file per table), a tiny file for each cell/field in a table. Then use zfs for versioning of data (or git). Zfs can already do software mirroring to drive pools on the machine, so maybe add rsync for remote pools. Optimize it for SSD's and you have a filesystem-database-replicating-versioned -based store. Maybe just need a nice cli/gui for zfs to make it easier/safer to expose more of it's interesting features.

zcw100 6y ago
“So we need a new tool that is basically git and rsync rolled into one.” That sounds a bit like IPFS

jiggawatts 6y ago

Windows Server has a variant of the Rsync algorithm that it uses for its Distributed File System Replication (DFS-R) that can look for up to 4 similar files. It works okay-ish in practice, and is better than nothing.