3 ms·

You are not using for long term archival though, it seems you are using it for deployment.

• latchkey 4y ago
  But the tradeoff between availability and integrity is different for data transmission than for data archiving. When transmitting data, usually the most important consideration is to avoid undetected errors (false negatives for corruption), because a retransmission can be requested if an error is detected. Archiving, on the other hand, usually implies that if a file is reported as corrupt, "retransmission" is not possible. Obtaining another copy of the file may be difficult or impossible. Therefore accuracy (freedom from mistakes) in the detection of errors becomes the most important consideration. Part of the issue with archival is transmission. I transmit these files over the internet (github->cloudflare->server) and I haven't seen a single failure of a file to unxz after transmission. This implies that I should see issues... but I really haven't.
  • jraph 4y ago
    Indeed. But if you transmit over TCP/IP, maybe SSL, probably over reliable links (github<->cloudflare<->server), seeing corruption is unlikely. And if unxz fails, in your case, you can just re-transmit. So using xz may work well for you and using it is not a big deal, but that does not make the format reliable for long term archival and as a sibling wrote, you probably could use lzip instead for the same benefits (and actually saving a few bytes because there are no extraneous headers) but none of the problems discussed in the article.