5 ms·

The first idea -- generating the reverse delta while applying the forward delta -- seem obvious. The second idea is clever, but Chrome did it in 2011: https://

The first idea -- generating the reverse delta while applying the forward delta -- seem obvious. The second idea is clever, but Chrome did it in 2011: https://news.ycombinator.com/item?id=2576878 https://news.ycombinator.com/item?id=2576878 Even if the ideas aren't novel, getting this thing built and rolled out for Windows is not at all trivial, so good job there. But calling it "new compression technology" is misleading.

willvarfar 5y ago
I desperately wish Linux distros used this trick too. I am tired of waiting for updates to download and having to clean out some cache folder that apt uses to free space.
- arianvanp 5y ago
  I think a lot of this work is being doing in projects like rpm-ostree
  - m4rtink 5y ago
    Yep ostree is doing delta updates by default. Flatpaks are alo stored as an ostree, so they do as well - even across all runtimes an flatpaks installed on a system! :)
- paulfurtado 5y ago
  FWIW, RPM based distros can use delta RPMs
- Saris 5y ago
  That's more an apt thing, dnf can use delta rpms with parallel downloads and automatically cleans unused packages. I'm not sure how well it cleans the cache though. On my fast home connection dnf is way faster than apt to update my laptop unless it's a really small update, then apt is faster because it refreshes the repos quicker. I think there are alternatives to apt that can do parallel downloads and delta as well.
- ahartmetz 5y ago
  Last time I looked (years ago), Fedora had delta updates - but updates were slow anyway.
- IceWreck 5y ago
  RPMs have delta updates
- secondaryacct 5y ago
  But at 50 to 100 MBytes/s on a multi Gb/s line I find it better, it's less the download than the time to install that bother me. Interestingly I found myself using ultra minimal distro vm, and installing nearly nothing on them (qubes os) and just shard the apps: the startup / update of the main gui linux is trivially fast, each vm only has a little subset of apps I care about, and when I trigger a many-vm update they do it in sequence and just update their own subset, meanwhile I can fully use the other vms. Also I tend to use stable-like distros now like debian and barely see updates coming in anymore, compared to my fedora youth when there was several updates a day, some breaking everything.
  - craftinator 5y ago
    > Interestingly I found myself using ultra minimal distro vm, and installing nearly nothing on them (qubes os) and just shard the apps Isn't this basically the idea behind snaps? Or is that more docker-like?
- xclap10 5y ago
  I cannot remember when a Debian update was slower than a Windows update. And that does not even mention Visual Studio updates, which pull Gigabytes of data. Debian updates are very fast.
  - Trex_Egg 5y ago
    Nowadays the visual studio updates are quite small for my workload with C++ tools.
  - tinus_hn 5y ago
    Detecting and installing a Debian update is much faster. But as they are much bigger whether this is offset by the download time depends on your bandwidth.
- cunidev 5y ago
  For what it's worth, Solus uses deltas since 2016. Updates tend to be amazingly fast. Shame that everything else in it is slowly going downhill.
- bmn__ 5y ago
  openSUSE (excluding Tumbleweed) had delta RPMs since 2005.
- deleted 5y ago
  [deleted]
exikyut 5y ago
Windows NT (as a usable thing available to the naive general public) didn't exist before Microsoft built it, so it was technically correct to call it new technology technology. In the same vein, this is arguably new compression technology. The above being said, my reaction to the article's key point was "you've been shipping forward and reverse diffs this entire time???" I guess my point is that it's kinda sad that it's most correct to generally describe things like this like as "new technology", because customers who've never seen something like it before don't apply any distinction between that and the idea having existed for however long. </rant>
- emteycz 5y ago
  Well of course the naive public doesn't make a distinction as it would be completely irrelevant to them. Can't use it now? Then it's a theory. A new thing I can buy today? Now that's innovative technology to them.
- ms4720 5y ago
  I always thought that it was just VMS done poorly
  - ch_123 5y ago
    The bits based on VMS (or more specifically the MICA project) were done quite well, it's all the Win32 crud on top which causes the problems.
  - rswail 5y ago
    The NT kernel is a nice design especially the (original) support for different flavors of syscalls to support both Windows and POSIX (and potentially others). A lot of that got lost over the years.
deleted 5y ago
[deleted]
mc32 5y ago
Have they improved the way the system figures out its inventory in order to know what patches it needs?
hcs 5y ago
For the second idea, updating references when the code moves is more directly used in a newer Chrome patch format: Zucchini [1]. Courgette does a complex process of disassembling, aligning labels, and sending a patch of the disassembly, so the patcher has to do disassembly, patching, and reassembly. Zucchini does a more straightforward disassembly pass to note where the references are, and then pre-patches them based on how it knows the code moved. Compared with Courgette, this involves less intermediate state so it's faster [2], the code is simpler, and because much more is implicit in the code motion the patches tend to be smaller. That basic technique goes back at least to the TranslateAddress method in Exediff (1999) [3]. I've been meaning to write up an article on this stuff [4], Google doesn't seem interested in publicizing Zucchini themselves, maybe due to the patent kerfuffle around Courgette. Microsoft's document on delta compression [5] covers a lot of this ground. [1] https://chromium.googlesource.com/chromium/src/+/HEAD/components/zucchini/README.md https://chromium.googlesource.com/chromium/src/+/HEAD/compon... [2] Some of this can be avoided, I made some changes to Courgette for a significant speed increase here: https://bugzilla.mozilla.org/show_bug.cgi?id=504624#c39 https://bugzilla.mozilla.org/show_bug.cgi?id=504624#c39 [3] http://robert.muth.org/Papers/1999-exediff.pdf http://robert.muth.org/Papers/1999-exediff.pdf [4] I did write up a bug to consider Zucchini in Firefox, with patch size comparisons, but ultimately we didn't switch from the simple power of bsdiff: https://bugzilla.mozilla.org/show_bug.cgi?id=1632374 https://bugzilla.mozilla.org/show_bug.cgi?id=1632374 [5] Their system can use info from the pre-linked objects and PDB symbol files for better alignment, I'd played around with seeding alignments like this in bsdiff and Zucchini but I don't recall it giving significant improvement. https://docs.microsoft.com/en-us/previous-versions/bb417345(v=msdn.10) https://docs.microsoft.com/en-us/previous-versions/bb417345(...

willvarfar 5y ago

I desperately wish Linux distros used this trick too. I am tired of waiting for updates to download and having to clean out some cache folder that apt uses to free space.

arianvanp 5y ago
I think a lot of this work is being doing in projects like rpm-ostree
- m4rtink 5y ago
  Yep ostree is doing delta updates by default. Flatpaks are alo stored as an ostree, so they do as well - even across all runtimes an flatpaks installed on a system! :)
paulfurtado 5y ago
FWIW, RPM based distros can use delta RPMs
Saris 5y ago
That's more an apt thing, dnf can use delta rpms with parallel downloads and automatically cleans unused packages. I'm not sure how well it cleans the cache though. On my fast home connection dnf is way faster than apt to update my laptop unless it's a really small update, then apt is faster because it refreshes the repos quicker. I think there are alternatives to apt that can do parallel downloads and delta as well.
ahartmetz 5y ago
Last time I looked (years ago), Fedora had delta updates - but updates were slow anyway.
IceWreck 5y ago
RPMs have delta updates
secondaryacct 5y ago
But at 50 to 100 MBytes/s on a multi Gb/s line I find it better, it's less the download than the time to install that bother me. Interestingly I found myself using ultra minimal distro vm, and installing nearly nothing on them (qubes os) and just shard the apps: the startup / update of the main gui linux is trivially fast, each vm only has a little subset of apps I care about, and when I trigger a many-vm update they do it in sequence and just update their own subset, meanwhile I can fully use the other vms. Also I tend to use stable-like distros now like debian and barely see updates coming in anymore, compared to my fedora youth when there was several updates a day, some breaking everything.
- craftinator 5y ago
  > Interestingly I found myself using ultra minimal distro vm, and installing nearly nothing on them (qubes os) and just shard the apps Isn't this basically the idea behind snaps? Or is that more docker-like?
xclap10 5y ago
I cannot remember when a Debian update was slower than a Windows update. And that does not even mention Visual Studio updates, which pull Gigabytes of data. Debian updates are very fast.
- Trex_Egg 5y ago
  Nowadays the visual studio updates are quite small for my workload with C++ tools.
- tinus_hn 5y ago
  Detecting and installing a Debian update is much faster. But as they are much bigger whether this is offset by the download time depends on your bandwidth.
cunidev 5y ago
For what it's worth, Solus uses deltas since 2016. Updates tend to be amazingly fast. Shame that everything else in it is slowly going downhill.
bmn__ 5y ago
openSUSE (excluding Tumbleweed) had delta RPMs since 2005.
deleted 5y ago
[deleted]

exikyut 5y ago

Windows NT (as a usable thing available to the naive general public) didn't exist before Microsoft built it, so it was technically correct to call it new technology technology. In the same vein, this is arguably new compression technology. The above being said, my reaction to the article's key point was "you've been shipping forward and reverse diffs this entire time???" I guess my point is that it's kinda sad that it's most correct to generally describe things like this like as "new technology", because customers who've never seen something like it before don't apply any distinction between that and the idea having existed for however long. </rant>

emteycz 5y ago
Well of course the naive public doesn't make a distinction as it would be completely irrelevant to them. Can't use it now? Then it's a theory. A new thing I can buy today? Now that's innovative technology to them.
ms4720 5y ago
I always thought that it was just VMS done poorly
- ch_123 5y ago
  The bits based on VMS (or more specifically the MICA project) were done quite well, it's all the Win32 crud on top which causes the problems.
- rswail 5y ago
  The NT kernel is a nice design especially the (original) support for different flavors of syscalls to support both Windows and POSIX (and potentially others). A lot of that got lost over the years.

deleted 5y ago

[deleted]

mc32 5y ago

Have they improved the way the system figures out its inventory in order to know what patches it needs?

hcs 5y ago

For the second idea, updating references when the code moves is more directly used in a newer Chrome patch format: Zucchini [1]. Courgette does a complex process of disassembling, aligning labels, and sending a patch of the disassembly, so the patcher has to do disassembly, patching, and reassembly. Zucchini does a more straightforward disassembly pass to note where the references are, and then pre-patches them based on how it knows the code moved. Compared with Courgette, this involves less intermediate state so it's faster [2], the code is simpler, and because much more is implicit in the code motion the patches tend to be smaller. That basic technique goes back at least to the TranslateAddress method in Exediff (1999) [3]. I've been meaning to write up an article on this stuff [4], Google doesn't seem interested in publicizing Zucchini themselves, maybe due to the patent kerfuffle around Courgette. Microsoft's document on delta compression [5] covers a lot of this ground. [1] https://chromium.googlesource.com/chromium/src/+/HEAD/components/zucchini/README.md https://chromium.googlesource.com/chromium/src/+/HEAD/compon... [2] Some of this can be avoided, I made some changes to Courgette for a significant speed increase here: https://bugzilla.mozilla.org/show_bug.cgi?id=504624#c39 https://bugzilla.mozilla.org/show_bug.cgi?id=504624#c39 [3] http://robert.muth.org/Papers/1999-exediff.pdf http://robert.muth.org/Papers/1999-exediff.pdf [4] I did write up a bug to consider Zucchini in Firefox, with patch size comparisons, but ultimately we didn't switch from the simple power of bsdiff: https://bugzilla.mozilla.org/show_bug.cgi?id=1632374 https://bugzilla.mozilla.org/show_bug.cgi?id=1632374 [5] Their system can use info from the pre-linked objects and PDB symbol files for better alignment, I'd played around with seeding alignments like this in bsdiff and Zucchini but I don't recall it giving significant improvement. https://docs.microsoft.com/en-us/previous-versions/bb417345(v=msdn.10) https://docs.microsoft.com/en-us/previous-versions/bb417345(...