3 ms·

I'm very curious; has a scheme that attempts to optimize against that worst case been considered? (Assuming you guys have, was it not used because the net trad

I'm very curious; has a scheme that attempts to optimize against that worst case been considered? (Assuming you guys have, was it not used because the net tradeoff wasn't worth it?) It seems like if you're already going through the effort to see if chunks have changed, you can readily do some heuristic where you check the end chunk of both sides, and scan inwards until you find a change; something to that respect, further minimize replicated data? I'd assume you might have CPU cycles to spare, but I'm too far into assuming already to feel comfortable so I'll just hope you answer :P (thanks in advance, it's been great to read what you've written thus far.)

brianwski 12y ago
> have you tried a better binary diff algorithm We just never got around to it. In practice, it turns out most well written programs with large data try not to insert one byte at the start of the file. For example, take your large Outlook "pst" file (commonly 1 - 4 GBytes). When you get an email, it seems to append it to the END of the pst file, plus update some internal tables. So a large amount of that won't change. Also, the worst case is it wastes space in our datacenter (and some bandwidth) for 30 days and then is cleaned up anyway, so you can measure the theoretical amount of money to save and it won't profoundly change our business so we put it off. Not to say ANYTHING is off the table, we're just always swamped with some project or other. :-)