3 ms·
I routinely distribute a ~5meg .xz (~20meg uncompressed) to 20k+ servers across multiple physical data centers on a regular basis. Haven't seen a single failure
by latchkey 4y ago
I routinely distribute a ~5meg .xz (~20meg uncompressed) to 20k+ servers across multiple physical data centers on a regular basis. Haven't seen a single failure. It ends up being 1.85megs smaller than the tgz version. Unless someone comes up with a better solution (ie: smaller), I probably won't change that any time soon.
- smaudet 4y agoWhat about lzip? The author claims the compression size is comparable or better.
- latchkey 4y agoI haven't tried that one yet. I'll play around and see what I can find out. 7zip didn't perform as well. tar c -C ./build $(BINARY) | gzip -9 - > $(PKG_NAME_GZ) tar c -C ./build $(BINARY) | xz -z -9e - > $(PKG_NAME)
- jraph 4y agoIt should, if the compression algorithm is actually the same, just the format specification that differs.
- exyi 4y agoIt will be the same, it also just uses LZMA. It has less headers, but that's not going to make much of a difference on 5M file
- latchkey 4y agoCompressed to 5mb. Original is about 20.
- jraph 4y agoYou are not using for long term archival though, it seems you are using it for deployment.
- latchkey 4y agoBut the tradeoff between availability and integrity is different for data transmission than for data archiving. When transmitting data, usually the most important consideration is to avoid undetected errors (false negatives for corruption), because a retransmission can be requested if an error is detected. Archiving, on the other hand, usually implies that if a file is reported as corrupt, "retransmission" is not possible. Obtaining another copy of the file may be difficult or impossible. Therefore accuracy (freedom from mistakes) in the detection of errors becomes the most important consideration. Part of the issue with archival is transmission. I transmit these files over the internet (github->cloudflare->server) and I haven't seen a single failure of a file to unxz after transmission. This implies that I should see issues... but I really haven't.
- jraph 4y agoIndeed. But if you transmit over TCP/IP, maybe SSL, probably over reliable links (github<->cloudflare<->server), seeing corruption is unlikely. And if unxz fails, in your case, you can just re-transmit. So using xz may work well for you and using it is not a big deal, but that does not make the format reliable for long term archival and as a sibling wrote, you probably could use lzip instead for the same benefits (and actually saving a few bytes because there are no extraneous headers) but none of the problems discussed in the article.
- jmillikin 4y agoYou may be interested in bsdiff[0] or Courgette[1]. If the file is a new version of what you deployed previously, you can use these programs to produce binary patch files. Diff the old and new binaries, and you'll only need to transmit the patch. bsdiff is a generic binary differ that is widely available. Courgette is optimized for executables and has an uncommon build system, but claims to produce files ~85% smaller than bsdiff. [0] https://www.daemonology.net/bsdiff/ https://www.daemonology.net/bsdiff/ [1] https://www.chromium.org/developers/design-documents/software-updates-courgette/ https://www.chromium.org/developers/design-documents/softwar...
- latchkey 4y agoTrue. The issue is that adds additional complexity on the receiving end that I didn't want to engineer for.
- espadrine 4y ago> Unless someone comes up with a better solution (ie: smaller) Be careful what you wish for! If you care about smaller above all else, there are much better compression schemes nowadays. NNCP is on the range of practical, and beats LZMA: https://bellard.org/nncp/ https://bellard.org/nncp/ If you really want to go crazy, large language models like BLOOM can be repurposed for compression; the Chinchilla paper lists a 0.3 bit-per-byte compression ratio on GitHub code. Of course, the cost is in GPU hardware, or in time.