4 ms·

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow f

by geokon 4y ago

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom diff algo? (if it's somehow not appropriate, it'd still be nice to port whatever secret sauce they use here to git..)

deathanatos 4y ago
I'm sort of in the same boat, but with the sentence, > To help this situation, we developed two tools, cdc_rsync Why not use rsync? (It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.) The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happened in the rsync case/GIF, but that wasn't the case considered, so it's an oranges/apples comparison. (Or, how is it that the local manages to be clairvoyant enough to choose the same windows?)
- mindwok 4y ago
  I was wondering the same thing. It isn't explicit in the write-up but it's because rsync does not have a native Windows implementation, only rsync under Cygwin, so they developed this to achieve the same thing except it turned out faster than rsync.
  - int_19h 4y ago
    MSYS2 has rsync. Unlike Cygwin, I would consider that native.
- ljusten 4y ago
  >> Or, how is it that the local manages to be clairvoyant enough to choose the same windows? Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.
mindwok 4y ago
They are separate problems and not really comparable, git is a higher level tool. The diffing in git is for humans to compare code, and as such it is actually computed on the fly and not stored anywhere (unless you use git patch). Similarly, git does not provide any mechanism for sending/receiving files, it re-uses existing solutions like HTTP and SSH. This tool exists to send/receive files, and the diffing is an implementation detail used to achieve a high level of performance. It would make more sense for git to use this library under the hood as you mentioned.
- geokon 4y ago
  Oh right, git doesn't store diffs. So each time you push/pull, it fetches/uploads the whole file.. for every commit? Damn.. Then I guess even using this under the hood would be essentially a rewrite or how git fundamentally works internally
  - pornel 4y ago
    There is git-pack which compresses similar files together.
  - rkangel 4y ago
    There is an analogy here, as git does deduplication 'under the hood'. It's kind of weird actually - at the architectural level we interact with, we talk in terms of diffs - both in terms of display and what we put in. At the next level down (content addressable store) git is storing whole files, and the git tooling translates the diffs we communicate about down into whole files for each commit. Then at the next level down, git puts files together in packfiles (when the repo is packed) which is a compression system to make use of the fact that most files are just tweaks of other files. So, once again it's diffs.
- planede 4y ago
  git does store deltas, but it's an implementation detail. This is what git pack files mostly are. I don't know if git sends or receives deltas though, I think it does?
desi_ninja 4y ago
Even if one tries to repurpose git for it, git doesn't work well with diffs in binaries

by geokon 4y ago

deathanatos 4y ago

I'm sort of in the same boat, but with the sentence, > To help this situation, we developed two tools, cdc_rsync Why not use rsync? (It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.) The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happened in the rsync case/GIF, but that wasn't the case considered, so it's an oranges/apples comparison. (Or, how is it that the local manages to be clairvoyant enough to choose the same windows?)

mindwok 4y ago
I was wondering the same thing. It isn't explicit in the write-up but it's because rsync does not have a native Windows implementation, only rsync under Cygwin, so they developed this to achieve the same thing except it turned out faster than rsync.
- int_19h 4y ago
  MSYS2 has rsync. Unlike Cygwin, I would consider that native.
ljusten 4y ago
>> Or, how is it that the local manages to be clairvoyant enough to choose the same windows? Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.

mindwok 4y ago

They are separate problems and not really comparable, git is a higher level tool. The diffing in git is for humans to compare code, and as such it is actually computed on the fly and not stored anywhere (unless you use git patch). Similarly, git does not provide any mechanism for sending/receiving files, it re-uses existing solutions like HTTP and SSH. This tool exists to send/receive files, and the diffing is an implementation detail used to achieve a high level of performance. It would make more sense for git to use this library under the hood as you mentioned.

geokon 4y ago
Oh right, git doesn't store diffs. So each time you push/pull, it fetches/uploads the whole file.. for every commit? Damn.. Then I guess even using this under the hood would be essentially a rewrite or how git fundamentally works internally
- pornel 4y ago
  There is git-pack which compresses similar files together.
- rkangel 4y ago
  There is an analogy here, as git does deduplication 'under the hood'. It's kind of weird actually - at the architectural level we interact with, we talk in terms of diffs - both in terms of display and what we put in. At the next level down (content addressable store) git is storing whole files, and the git tooling translates the diffs we communicate about down into whole files for each commit. Then at the next level down, git puts files together in packfiles (when the repo is packed) which is a compression system to make use of the fact that most files are just tweaks of other files. So, once again it's diffs.
planede 4y ago
git does store deltas, but it's an implementation detail. This is what git pack files mostly are. I don't know if git sends or receives deltas though, I think it does?

desi_ninja 4y ago

Even if one tries to repurpose git for it, git doesn't work well with diffs in binaries