3 ms·

I had to make an archiver once (commercial) so I did think about it a bit. I am not sure pack would solve anything for me. It obviously solves the authors usec

by t43562 3y ago

I had to make an archiver once (commercial) so I did think about it a bit. I am not sure pack would solve anything for me. It obviously solves the authors usecases but tar has some tricks which I don't want to lose; * Able to write to a pipe/socket - lets you not waste space or time by writing to disc something that you intend to transmit over a pipe or TCP socket anyhow. It's almost a "virtual" archive and it should be possible to make one that is far too big to fit into memory - because as you send each bit of it you deallocate that memory. At the receiver each bit can be written to disc or extracted and then that memory is reused for the next bit - so the archive never fully "exists" but it does the job of serialising some data. An example could be piping the output of tar to an ssh command which untars it on a remote machine. * Metadata has to be with the file data - not stuck at the end of the file - because you need to be able to start work without waiting till the file is fully received through your pipe. You don't want to be forced to have space to store the archive and the extracted files (may be a huge archive). * Choice of compression - lzop is super fast such that using it can sometimes give slightly better performance than writing uncompressed data. OTOH that might not be your concern and XZ might suit you by compressing much more thoroughly. Either way it's very nice to have compression that works across multiple files - which is especially helpful when compressing a lot of small files such as source code. * Ability to encapsulate - should be able to put the packed data into any imaginable container like an encryption or data transmission protocol without insisting that the entire archive has to be fully read before members can start to be extracted/processed. This is essentially the same as the pipe/socket requirement. I'm not saying that these things matter to everyone - I have just found them incredibly useful in a few critical situations. The world of ZIP users on Windows seems to be sort of blind to them - thinking firmly in that box.

masklinn 3y ago
> The world of ZIP users on Windows seems to be sort of blind to them - thinking firmly in that box. Seems to me like this is you being blind to certain use cases, and so stuck in your streaming oriented box that you can not conceive of other use cases where a streaming format is actively detrimental.
- t43562 3y ago
  I started on ZIP like most people and discovered what could be done with streaming so I don't think so.
OttoCoddo 3y ago
Hey fellow enthusiast. - Piping is really easy and it will get added to Pack. It is matter of time, until these features get added as they will be added based on popularity and piping is not that popular for most people. But I get you and I will add it for you. - Metadata is not stored in Pack. I don’t want the metadata of my machine attached to a file. It’s a never-ending nightmare to match source and destination OS metadata. There will always be something missing, and Pack tends to get everything perfect or nothing. Storing metadata adds extra weight that most users don’t care about and complicates the ability to store other types of data alongside files. It may get added as an option in the future if many people need it. - Pack uses Zstandard under the hood. Great compression speed and ratio. In my opinion, it is the leading algorithm in the field and makes it a proper choice to use instead of DEFLATE, used in ZIP or GZIP. - At this point you are telling tar features. tar is not random access, Pack is. As an example, if I want to extract just a .c file from the whole codebase of Linux, it can be done (on my machine) in 30 ms, compared to near 500 ms for WinRAR or 2500 ms for tar.gz. And it will just worsen when you count encryption. For now, Pack encryption is not public, but when it is, you can access a file in a locked Pack file in a matter of milliseconds rather than seconds.
- t43562 3y ago
  I think random access is a useful feature - e.g. if you want to compress code modules like java does so that you can still load individual modules quickly. This is not something I've wanted yet personally but that's just random chance. When I do need it I will know which tool to use! Thanks. It could be handy to be able to mount a pack like a filesystem.
  - OttoCoddo 3y ago
    Indeed. Benchmarking Pack as a file system was fun. It is near 10X faster to let you iterate all the files compared to what I get from NTFS (warm with all caching on for both solutions). Someday, it can be used as a virtual drive. I leave it to future people.

by t43562 3y ago

masklinn 3y ago

> The world of ZIP users on Windows seems to be sort of blind to them - thinking firmly in that box. Seems to me like this is you being blind to certain use cases, and so stuck in your streaming oriented box that you can not conceive of other use cases where a streaming format is actively detrimental.

t43562 3y ago
I started on ZIP like most people and discovered what could be done with streaming so I don't think so.

OttoCoddo 3y ago

Hey fellow enthusiast. - Piping is really easy and it will get added to Pack. It is matter of time, until these features get added as they will be added based on popularity and piping is not that popular for most people. But I get you and I will add it for you. - Metadata is not stored in Pack. I don’t want the metadata of my machine attached to a file. It’s a never-ending nightmare to match source and destination OS metadata. There will always be something missing, and Pack tends to get everything perfect or nothing. Storing metadata adds extra weight that most users don’t care about and complicates the ability to store other types of data alongside files. It may get added as an option in the future if many people need it. - Pack uses Zstandard under the hood. Great compression speed and ratio. In my opinion, it is the leading algorithm in the field and makes it a proper choice to use instead of DEFLATE, used in ZIP or GZIP. - At this point you are telling tar features. tar is not random access, Pack is. As an example, if I want to extract just a .c file from the whole codebase of Linux, it can be done (on my machine) in 30 ms, compared to near 500 ms for WinRAR or 2500 ms for tar.gz. And it will just worsen when you count encryption. For now, Pack encryption is not public, but when it is, you can access a file in a locked Pack file in a matter of milliseconds rather than seconds.

t43562 3y ago
I think random access is a useful feature - e.g. if you want to compress code modules like java does so that you can still load individual modules quickly. This is not something I've wanted yet personally but that's just random chance. When I do need it I will know which tool to use! Thanks. It could be handy to be able to mount a pack like a filesystem.
- OttoCoddo 3y ago
  Indeed. Benchmarking Pack as a file system was fun. It is near 10X faster to let you iterate all the files compared to what I get from NTFS (warm with all caching on for both solutions). Someday, it can be used as a virtual drive. I leave it to future people.