4 ms·

Dropbox uses very sophisticated binary diff nobody else provides to minimize inbound/outbound traffic; this has some CPU/battery cost though. Competitors usuall

by bitL 7y ago

Dropbox uses very sophisticated binary diff nobody else provides to minimize inbound/outbound traffic; this has some CPU/battery cost though. Competitors usually provide very dumb upload/download that might be less CPU/battery intense but much heavier on the bytes transferred. You simply might not be their target group.

deanCommie 7y ago
Which may be a feature that belongs in the bygone era of lower network speeds and bandwidth limits.
- bitL 7y ago
  Do you like higher bills on your iPhone when accessing your shared data? Not sure why would efficiency belong to bygone era. I'd rather expect CPU to handle this diff process with some dedicated instructions more efficiently in the future.
anyfoo 7y ago
I genuinely don't know, but are you sure that others don't provide this?
- bitL 7y ago
  GDrive uses something internally but they don't expose it to public clients, so do they really provide it? (I might be outdated here, it's been a while I last checked)
tbrock 7y ago
Yeah, I don't think so. Their target group must be people with 10mb of files or less. Why would you need to binary diff when syncing dropbox to a new computer? Yet setting up dropbox for the first time on a new machine uses 100% of my CPU. The same S3 sync operation is barely noticible. When updating files, first check for changes with something ultra cheap like an md5 then, if changed, do the "sophisticated" stuff.
- guitarbill 7y ago
  onboarding cost might be annoying, but at least it's upfront. i'd take 100% at the start for more efficient sync later.
  - tbrock 7y ago
    Yeah, if only it did that...
- bitL 7y ago
  They need to store your data somewhere, they need to be able to reuse/serve chunks from other users that share the same file to make its storage cost efficient etc.
function_seven 7y ago
I very much doubt that Dropbox's target group are users with bandwidth limitations. Yes, it's nice to have features that minimize waste on the network link, but I'd bet far more people would trade that away in exchange for laptop battery life and a smaller hit on their CPU. If they're only targeting that niche, then they have a business model problem. Make it an option ("Minimize bandwidth or battery usage?")
manicdee 7y ago
Those who do not study ~~history~~ ~~Unix~~ rsync are doomed to repeat it poorly.
- bitL 7y ago
  I used rsync to mirror one encrypted 14TB drive to another one. It took ~2 weeks. I wouldn't call rsync "good enough".
  - manicdee 7y ago
    What was the bottleneck?
    - mynegation 7y ago
      I am not bitl but my guesses, in order of probability would be: * using filesystem that is not natively supported on one or both drives, like fuse ntfs on Linux or ext3 on Mac. * copying over slower network like wifi * lots and lots of small files especially in a single directory * usb 2.0
  - michaelcampbell 7y ago
    Something is very, very wrong then. rsync will use 100% of the bandwidth provided. It won't be any slower than anything else you might try over the same network. If you found that to be the case, it's almost certainly something else.
    - bitL 7y ago
      I guess if you have many overlapping files on both drives and you force rsync to actually look inside and compare them, its efficiency might suffer? It wasn't just a simple copy from one full drive to another empty one, rather two drives with overlapping files that had to be synced. I experimented with rsync to see if I could replace RAID with it, i.e. one primary drive, one backup drive synced on the background instead of mirroring directly (and getting the same low-level bugs spread to both drives or getting both drives to die at around the same time). Both drives were encrypted on the device level (dm-crypt AES128) with standard ext4 on top of mapped drive. I previously ran configs with RAID-1 as well as BTRFS RAID-5 (horribly buggy and dangerous) and wanted to see if rsync makes sense for this purpose (it doesn't). I couldn't use ZFS as that NAS didn't have ECC.

by bitL 7y ago

deanCommie 7y ago

Which may be a feature that belongs in the bygone era of lower network speeds and bandwidth limits.

bitL 7y ago
Do you like higher bills on your iPhone when accessing your shared data? Not sure why would efficiency belong to bygone era. I'd rather expect CPU to handle this diff process with some dedicated instructions more efficiently in the future.

anyfoo 7y ago

I genuinely don't know, but are you sure that others don't provide this?

bitL 7y ago
GDrive uses something internally but they don't expose it to public clients, so do they really provide it? (I might be outdated here, it's been a while I last checked)

tbrock 7y ago

Yeah, I don't think so. Their target group must be people with 10mb of files or less. Why would you need to binary diff when syncing dropbox to a new computer? Yet setting up dropbox for the first time on a new machine uses 100% of my CPU. The same S3 sync operation is barely noticible. When updating files, first check for changes with something ultra cheap like an md5 then, if changed, do the "sophisticated" stuff.

guitarbill 7y ago
onboarding cost might be annoying, but at least it's upfront. i'd take 100% at the start for more efficient sync later.
- tbrock 7y ago
  Yeah, if only it did that...
bitL 7y ago
They need to store your data somewhere, they need to be able to reuse/serve chunks from other users that share the same file to make its storage cost efficient etc.

function_seven 7y ago

I very much doubt that Dropbox's target group are users with bandwidth limitations. Yes, it's nice to have features that minimize waste on the network link, but I'd bet far more people would trade that away in exchange for laptop battery life and a smaller hit on their CPU. If they're only targeting that niche, then they have a business model problem. Make it an option ("Minimize bandwidth or battery usage?")

manicdee 7y ago

Those who do not study ~~history~~ ~~Unix~~ rsync are doomed to repeat it poorly.

bitL 7y ago
I used rsync to mirror one encrypted 14TB drive to another one. It took ~2 weeks. I wouldn't call rsync "good enough".
- manicdee 7y ago
  What was the bottleneck?
  - mynegation 7y ago
    I am not bitl but my guesses, in order of probability would be: * using filesystem that is not natively supported on one or both drives, like fuse ntfs on Linux or ext3 on Mac. * copying over slower network like wifi * lots and lots of small files especially in a single directory * usb 2.0
- michaelcampbell 7y ago
  Something is very, very wrong then. rsync will use 100% of the bandwidth provided. It won't be any slower than anything else you might try over the same network. If you found that to be the case, it's almost certainly something else.
  - bitL 7y ago
    I guess if you have many overlapping files on both drives and you force rsync to actually look inside and compare them, its efficiency might suffer? It wasn't just a simple copy from one full drive to another empty one, rather two drives with overlapping files that had to be synced. I experimented with rsync to see if I could replace RAID with it, i.e. one primary drive, one backup drive synced on the background instead of mirroring directly (and getting the same low-level bugs spread to both drives or getting both drives to die at around the same time). Both drives were encrypted on the device level (dm-crypt AES128) with standard ext4 on top of mapped drive. I previously ran configs with RAID-1 as well as BTRFS RAID-5 (horribly buggy and dangerous) and wanted to see if rsync makes sense for this purpose (it doesn't). I couldn't use ZFS as that NAS didn't have ECC.