5 ms·
I'm not sure if tar is a good format for backups, due to its extremely sequential nature (e.g. you can't get a list of the files in an archive without scanning
by boramalper 9y ago
I'm not sure if tar is a good format for backups, due to its extremely sequential nature (e.g. you can't get a list of the files in an archive without scanning the whole archive...)
- chrisseaton 9y agoWhat overhead does that add in practice? Like a second or so to scan a few GB of tarball? Does that matter?
- ctur 9y agoIt often is, especially if the tarball is compressed (especially if you use anything but Zstandard) or on a network store rather than fast local ssd. Typically a rule of thumb is a hard drive with a spindle may five around 100mb/sec throughput, so that is likely the bigger bottleneck, followed by cpu if using slower compression algorithms. Zstandard at a high compression level gives a good tradeoff of decompression speed vs underlying physical throughput, be it network or spindle. Even so, yes, the sequential nature of tar is not great for this reason. Squashfs files, which can be used on Linux and Mac (using fuse for OS X), are a very good alternative for archIves, though without some of the features described in the linked article.
- boramalper 9y agoI guess it can be cumbersome as well if there are many small* files (since you need to iterate over all of them) until you find the one you are looking for. Think of it like a linked list. *: As opposed to a few large files.
- noselasd 9y agoyou need to download the entire thing to scan it. if it already resides locally on a seekable media , there's litle drawback
- Crontab 9y agoThis issue occurred to me as well (especially for encrypted tarballs). My workaround was to create a listing of the files that accompanies the tarball, such as: "backup.tar" and "backup.list".