12 ms·

I don't understand what bsdiff does, or is. I am a software developer and I frankly have no clue what I would ever use bsdiff for! I've read what it does (lib

I don't understand what bsdiff does, or is. I am a software developer and I frankly have no clue what I would ever use bsdiff for! I've read what it does (libraries for building and applying patches to binary files) and still don't really have a sense for what the purpose of this tool is. What are some real life use cases for it? When does a developer need such a tool?

deleted 5y ago
[deleted]
faeyanpiraat 5y ago
If you have a game, where you have large packs of files (like a 2GB sized textures.pak), and in a patch you want to add 2 small things, you can just ship the differencd between the old anc new pack file, and skip transferring the rest.
- OldTimeCoffee 5y ago
  They do this, but not as a binary diff. The new files are just shipped as an override of the old files that are loaded after the old files. It's substainally faster than applying a binary diff and file sizes really aren't a concern for the most part. The game industry really hasn't used binary diffs and patching since the 90s when they used RTPatch.
  - mcguire 5y ago
    I've noticed. Flight Simulator downloads tens of GB every time I start it. :-/
throwaway3699 5y ago
Implementing software updates where you don't want to ship entire binaries again (and only the diff) would be one. In some video games the assets are also packed into massive binaries, so you don't want to ship gigabytes of data because you replaced one icon. Sadly many games do this anyway nowadays.
- maccard 5y ago
  One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Also, as a dev, you have no idea what version your users are updating _from_. You either need to generate some number of patches for every version you could be updating from, and figure out if you should just download the whole thing again in any of those cases anyway.
  - oblio 5y ago
    > One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Does this happen with more advanced compression algorithms? I've rsynced zip files of different versions of internal software and the diff was always much, much smaller than the entire package.
    - whelming_wave 5y ago
      Kind of an aside from your question, but binary patches not being much smaller than the full thing might happen more with modern games? Heresay, but from what I've heard modern games may ship multiple copies of some assets with different levels or features so they can be loaded as a sequential read off the disk. While a block-oriented compression algorithm might sync up more reliably, if you're packing 200MB of assets for a level and they're all compressed to take advantage of the fact they'll be read sequentially could mean a change 25MB in would still ship ~175MB of changes.
      - uuidgen 5y ago
        So far in in most of the world even platter disks (which have really poor performance with modern Windows) are faster than network. Which means you can download description of the difference and reorder the file locally much faster than downloading it. Yes it needs the file to made in a way that is update friendly - most current compression algorithms can be configured like that. Yes, compression will be slightly lower, but you will save on both download size AND disk space, because right now most patches require you to download patch and then apply it, requiring twice the space. If you have 5% larger asset file but can patch it with a few memcpy on a mapped file it is a win in every way imaginable. It is just really poor programming, nothing more. And it's everywhere. If find source >/dev/null takes 6 seconds there is no reason for gradle to take 2 minutes on a rebuild. If the dev is used to that, why would they even think about patch optimisation?
cperciva 5y ago
Does it help if I tell you that bsdiff has saved end users somewhere over 10,000 years of waiting for software updates to download?
- throw14082020 5y ago
  yea well thats not enough! Xcode should be using it too. They make us download 12GB for every update. /s end open source entitlement section Thanks for your great work
nix23 5y ago
Chrome and Chromium used bsdiff for updates: https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...
- codeulike 5y ago
  The page says they use something called Courgette which is similar to bsdiff
  - nix23 5y ago
    They used bsdiff before and now Courgette.
codeulike 5y ago
Its old now but here's a description from Chromium about something they use called Courgette which is similar/related to bsdiff https://blog.chromium.org/2009/07/smaller-is-faster-and-safer-too.html https://blog.chromium.org/2009/07/smaller-is-faster-and-safe... The explanation here is pretty fascinating https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...
- kevincox 5y ago
  The really cool TL;DR here is that courgette "disassembles" the binary before diffing. Basically turning internal references into symbolic references. This way adding an extra instruction to a function won't affect all of the relative addresses in surrounding code.
masklinn 5y ago
> I don't understand what bsdiff does, or is. It‘s a binary diff/patch utility. > What are some real life use cases for it? When does a developer need such a tool? Incrementally update binary files e.g. assets or runnable binaries instead if having to re-send the entire thing on every update so e.g. games, browsers, package managers, … The standard diff/patch utilities are text-based and even when they do extend to binary data their algorithms and heuristics tend to be biased towards textual contents. Bsdiff was built specifically with an eye towards executables.

deleted 5y ago

[deleted]

faeyanpiraat 5y ago

If you have a game, where you have large packs of files (like a 2GB sized textures.pak), and in a patch you want to add 2 small things, you can just ship the differencd between the old anc new pack file, and skip transferring the rest.

OldTimeCoffee 5y ago
They do this, but not as a binary diff. The new files are just shipped as an override of the old files that are loaded after the old files. It's substainally faster than applying a binary diff and file sizes really aren't a concern for the most part. The game industry really hasn't used binary diffs and patching since the 90s when they used RTPatch.
- mcguire 5y ago
  I've noticed. Flight Simulator downloads tens of GB every time I start it. :-/

throwaway3699 5y ago

Implementing software updates where you don't want to ship entire binaries again (and only the diff) would be one. In some video games the assets are also packed into massive binaries, so you don't want to ship gigabytes of data because you replaced one icon. Sadly many games do this anyway nowadays.

maccard 5y ago
One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Also, as a dev, you have no idea what version your users are updating _from_. You either need to generate some number of patches for every version you could be updating from, and figure out if you should just download the whole thing again in any of those cases anyway.
- oblio 5y ago
  > One of the reasons games do this is the data is compressed, so a "patch" might be indistinguishable from a real update. Does this happen with more advanced compression algorithms? I've rsynced zip files of different versions of internal software and the diff was always much, much smaller than the entire package.
  - whelming_wave 5y ago
    Kind of an aside from your question, but binary patches not being much smaller than the full thing might happen more with modern games? Heresay, but from what I've heard modern games may ship multiple copies of some assets with different levels or features so they can be loaded as a sequential read off the disk. While a block-oriented compression algorithm might sync up more reliably, if you're packing 200MB of assets for a level and they're all compressed to take advantage of the fact they'll be read sequentially could mean a change 25MB in would still ship ~175MB of changes.
    - uuidgen 5y ago
      So far in in most of the world even platter disks (which have really poor performance with modern Windows) are faster than network. Which means you can download description of the difference and reorder the file locally much faster than downloading it. Yes it needs the file to made in a way that is update friendly - most current compression algorithms can be configured like that. Yes, compression will be slightly lower, but you will save on both download size AND disk space, because right now most patches require you to download patch and then apply it, requiring twice the space. If you have 5% larger asset file but can patch it with a few memcpy on a mapped file it is a win in every way imaginable. It is just really poor programming, nothing more. And it's everywhere. If find source >/dev/null takes 6 seconds there is no reason for gradle to take 2 minutes on a rebuild. If the dev is used to that, why would they even think about patch optimisation?

cperciva 5y ago

Does it help if I tell you that bsdiff has saved end users somewhere over 10,000 years of waiting for software updates to download?

throw14082020 5y ago
yea well thats not enough! Xcode should be using it too. They make us download 12GB for every update. /s end open source entitlement section Thanks for your great work

nix23 5y ago

Chrome and Chromium used bsdiff for updates: https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...

codeulike 5y ago
The page says they use something called Courgette which is similar to bsdiff
- nix23 5y ago
  They used bsdiff before and now Courgette.

codeulike 5y ago

Its old now but here's a description from Chromium about something they use called Courgette which is similar/related to bsdiff https://blog.chromium.org/2009/07/smaller-is-faster-and-safer-too.html https://blog.chromium.org/2009/07/smaller-is-faster-and-safe... The explanation here is pretty fascinating https://www.chromium.org/developers/design-documents/software-updates-courgette https://www.chromium.org/developers/design-documents/softwar...

kevincox 5y ago
The really cool TL;DR here is that courgette "disassembles" the binary before diffing. Basically turning internal references into symbolic references. This way adding an extra instruction to a function won't affect all of the relative addresses in surrounding code.

masklinn 5y ago

> I don't understand what bsdiff does, or is. It‘s a binary diff/patch utility. > What are some real life use cases for it? When does a developer need such a tool? Incrementally update binary files e.g. assets or runnable binaries instead if having to re-send the entire thing on every update so e.g. games, browsers, package managers, … The standard diff/patch utilities are text-based and even when they do extend to binary data their algorithms and heuristics tend to be biased towards textual contents. Bsdiff was built specifically with an eye towards executables.