5 ms·
It was hard to believe for me, too. And I didn't stumble upon it; I looked for it closely, and that was a point in the note. People did not look properly for ne
by OttoCoddo 3y ago
It was hard to believe for me, too. And I didn't stumble upon it; I looked for it closely, and that was a point in the note. People did not look properly for nearly three decades. Many things have changed, but we computer people are still using the same tools.
I am not saying old is not good; the current solutions are great, but what are we, if we don't look for the better?
Yes it is that much faster, and a good part of it is because of the multi-thread design, but as a reminder, WinRAR or 7-Zip are too multi-thread, and you can see the difference.
To satisfy your doubt, I suggest running Pack for yourself. I am looking for more data on its behaviour on different machines and data.
Can I ask why do you need a version without ZSTD? If you are thinking that compression slows it down, I should say no. Pack is the first of its kinds that "Store" is slowing it down. Because its compression is smart, it will skip any non-compressible content.
On the same machine and the same Linux source code test:
Pack: 194 MB, 1.3 s
Pack (With no Press): 1.25 GB, 1.8 s
- out_of_protocol 3y agoPure zstd (or .tar.zstd) vs pack vs patched 7z+zstd would be more interesting, how much overhead introduced by pack format itself - in size and speed
- OttoCoddo 3y agoI answered this question here: https://news.ycombinator.com/item?id=39801083 https://news.ycombinator.com/item?id=39801083 If that is not enough, let me know.
- out_of_protocol 3y agotar.zst vs pack is looking great, thanks! Also there is https://github.com/mcmilk/7-Zip-zstd https://github.com/mcmilk/7-Zip-zstd .pack vs zst-7z with the same compression settings would b interesting. That will be pure container overhead
- xcdzvyn 3y agoMy concern with Pack obliging me to compress is that compression becomes less pluggable; I'd much rather my archive format be agnostic of compression, as with tar, so that I can trivially move to a better compression format when one inevitably comes to be.
- OttoCoddo 3y agoYou got a point. Although with that that option comes a great cost: We will lose portability, speed and even reliability. Portability: Receiver (or future you) needs to know what you used, and what version even. Speed: If you want to do the archive part first (tar) and then compress (gz), you will get much lower speed (as shown in the note). Reliability: Most people use tar with gz anyway, but if you use it with not so popular algorithm and tools, you will risk having a file that may or may not work into the future. Pack plan is to use the best of time (Zstandard) and if an update is needed in years to come, it will add support for the new algorithm updates. All Pack clients must only write the latest version (and read all previous versions) and that makes sure almost all use the best of their time.