12 ms·
Hop: Faster than unzip and tar at reading individual files
- nathell 5y agoRelated: pixz – a variant of xz that works with tar and enables (semi)efficient random access: https://github.com/vasi/pixz https://github.com/vasi/pixz
- CannoloBlahnik 5y agoThis seems like more of a tar problem than a zip problem, unless I'm missing something, given the lack of compression on Hop.
- slaymaker1907 5y agoI think you can still zip files without compression.
- sigzero 5y agoYes, but I don't think anyone does that.
- masklinn 5y agoThis ability is an important aspect of “document” archives following the open container format (OCF): the first file in the archive must be uncompressed, called “mimetypes”, and contain the mimetype of the document encoded in us-ascii.
- sumtechguy 5y agoUsually the -0 option. https://linux.die.net/man/1/zip https://linux.die.net/man/1/zip -mx=0 for 7zip style programs. If you also used the mt option with 7zip and just stored you probably could have a decent read rate as I think it spins extra threads.
- yboris 5y agoSince Hop doesn't do compression, the most appropriate comparison would be to asar https://github.com/electron/asar https://github.com/electron/asar It's not hard being faster than zip if you are not compressing/uncompressing.
- nightcracker 5y ago> It's not hard being faster than zip if you are not compressing/uncompressing Actually, since CPUs are so fast and disk IO is so slow, it is essentially impossible to beat a properly tuned program that reads data from disk and decompresses it on the fly using one that doesn't use compression at all.
- gnabgib 5y agotar doesn't do compression either, and zip doesn't NEED to (several file formats are just bundles of files in a zip with/without compression)
- syrrim 5y agoTar does do compression, via the standard -z flag. Every tar I have ever downloaded used some form of compression, so its hardly an optional part of the format.
- nemetroid 5y agoTar (the tool) does compression. Tar (the format) does not. Compression is applied separately from archiving. https://www.gnu.org/software/tar/manual/html_node/Standard.html https://www.gnu.org/software/tar/manual/html_node/Standard.h...
- codetrotter 5y agoIt is important to distinguish between tar the format and tar the utility. tar the utility is a program that can produce tar files, but it is also able to then compress that file. When you produce a compressed tar file, the contents are written into a tar file, and this tar file as a whole is compressed. Sometimes the compressed files are named in full like .tar.gz, .tar.bz2 or .tar.xz but often they are named as .tgz, .tbz or .txz respectively. In a way you could consider those files a format in their own right, but at the same time they really are simply plain tar files with compression applied on top. You can confirm this by decompressing the file without extracting the inner tar file using gunzip, bunzip2 or unxz respectively. This will give you a plain tar file as a result, which is a file and a format of its own. You can also see the description of compressed tar files with the “file” command and it will say for example “xz compressed data” for a .txz or .tar.xz file (assuming of course that the actual data in the file is really this kind of data). And a plain uncompressed tar file described by “file” will say something like “POSIX tar archive”.
- bouk 5y agoI wonder how this format compares to SquashFS
- 1MachineElf 5y agoIs there anything like this for RAR files? I'm looking for an alternative to unrar, as I've recently learned that it's code is actually non-free.
- BlackLotus89 5y agoI think maybe you misunderstood this project.... Anyway libarchive can read rar files so just use bsdtar. It can do many other archive files [0] like cpio as well. One standardized interface for everything is nice [0] https://github.com/libarchive/libarchive#supported-formats https://github.com/libarchive/libarchive#supported-formats
- abetusk 5y agoThere exists a utility called tarindexer [0] that can be used for random access to tar files. An index text file is created (one time) that is used to record the position of the files in the tar archive. Random reads are done by loading the index file and then seeking to the location of the file in question. For random access to gzip'd files, bgzip [1] can be used. bgzip also uses an index file (one time creation) that is used to record key points for random access. [0] https://github.com/devsnd/tarindexer https://github.com/devsnd/tarindexer [1] http://www.htslib.org/doc/bgzip.html http://www.htslib.org/doc/bgzip.html
- selfhoster11 5y agoThat just sounds like an inferior version of ZIP. IMO, unless you can only work with the tar format (which is a perfectly valid explanation, some programs are long-lived), ZIP is a better option for seekable archives because it supports even LZMA compression.
- lazide 5y agoZip is….. weird. In some undesirably ways sometimes, and desirable in others. The index is stored at the tail end of a zip file for instance, which is really not cool for something likej tape, doubly not cool when you don’t know in advance how big the data on the tape is.
- selfhoster11 5y agoDon't tapes support some form of seeking?
- lazide 5y agoThey do - it’s very slow, and entirely linear. It also puts wear on the tape, so if you do it a lot you’ll break the tape in a not-super-long-time. And since you wouldn’t know where the end is by reading the beginning, you’ll have to keep reading until you hit the right marker - and seek backwards (which you generally can’t read backwards on most tape drives, so you need to jump back and re-read).
- mattfrommars 5y agoIs 7zip = zip ?
- wolpoli 5y agoNo. The 7zip format has much better compression than the ancient (1990s) zip format.
- selfhoster11 5y agoZIP as a standard is no longer ancient. It has support for modern encryption and compression, including LZMA. This is sometimes referred to as the ZIPX archive, but it's part of the standard in its later revisions.
- mjevans 5y agozipx != zip -- we are speaking of file compression standards NOT branded compression software programs
- selfhoster11 5y agoNo, it literally is part of the ZIP standard [0]. I updated my original comment. ZIPX archives are simply ZIP archives that have been branded with the X for the benefit of the users who may be trying to open them with something ancient, that doesn't yet support LZMA or the other new features. [0] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
- mjevans 5y agoWhen a 'hacker' (programmer) says 'zip file' they refer to the widely compatible implementation which is likely to work in any common implementation of archive handling software. https://en.wikipedia.org/wiki/ZIP_(file_format)#Version_history https://en.wikipedia.org/wiki/ZIP_(file_format)#Version_hist... According to the above version table, that would conform to PKWARE 2.0, which as with the ODF file specification, limits packing methods to STORE and DEFLATE only; as mentioned in the standardization section and ISO/IEC 21320-1. When you say 'zip' you should be saying 'zipx' at the very least, but you may as well be comparing a tar file to a tar.zstd file in the same case.
- tyingq 5y agoAlso see SQLite archive files. Random access and compression. https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html
- dan-robertson 5y agoYeah I wonder how the format compares to this. It also wasn’t obvious whether hop supports adding data with an operation that isn’t just append (a stricter requirement for tar because it was designed for magnetic tape). It wasn’t clear to me from the table. sqlar can add to an existing archive and doesn’t require uncommon software to extract if you don’t have the right tool to hand.
- seeekr 5y agoFrom the README: "Why? Reading and writing lots of tiny files incurs significant syscall overhead, and (npm) packages often have lots of tiny files. (...)" It seems the author is working on a fast JS bundler tool (https://bun.sh https://bun.sh) and the submission links to an attempt to fix some of the challenges in processing lots of npm files quickly. (But of course could be useful beyond that.)
- jraph 5y ago> "Why? Reading and writing lots of tiny files incurs significant syscall overhead, and (npm) packages often have lots of tiny files" Ah, those trees of is-character packages depending on is-letter-a packages, themselves depending on is-not-number packages each appearing several times in different versions are probably challenging to bundle and unpack efficiently. We might want to address file systems too, so they too can handle NPM efficiently (actual size on disk and speed). Or maybe the actual fix is elsewhere. Runs, fleeing their screen screaming in fear and frustration (no offense to the author, I actually find the problem interesting and the work useful)
- aaaaaaaaaaab 5y agoObviously, what we really need is an OS optimized for npm.
- zkldi 5y agoIf I had asked them what they wanted, they would've said faster filesystems.
- eyelidlessness 5y agoCame to mention Bun when I saw this hit the front page. I’ve been following Jarred’s Twitter since I heard about Bun and it’s quite impressive (albeit incomplete). To folks wondering why another bundler/what makes Bun special: - Faster than ESBuild/SWC - Fast build-time macros written as JSX (likely friendlier to develop than say a Babel plugin/macro). These open up a lot of possibilities that could benefit end users too, by performing more work on build/server and less client side. - Targeting ecosystem compatibility (eg will probably support the new JSX transform, which ESBuild does not and may not in the future) - Support for integration with frameworks, eg Next.js - Other cool performance-focused tools like Hop and Peechy[1] (though that’s a fork of ESBuild creator’s project Kiwi) This focus on performance is good for the JS ecosystem and for the web generally. 1: https://github.com/Jarred-Sumner/peechy https://github.com/Jarred-Sumner/peechy
- notananthem 5y ago(cries in winrar)
- Octplane 5y agoHey! Did you register me? </nag>
- kazinator 5y ago> Can't be larger than 4 GB How does that even happen in 2021?
- maccard 5y ago4GB is the maximum value of a 32 bit unsigned int. If I had to guess that's the maximum size of the array/vector container in zig.
- hansvm 5y agoZig has weird limits in a few places (1k compile-time branches without additional configuration, the width of any integer type needs to be less than 2^16, ...). Array/vector length's aren't the issue though; you can happily work with 64-bit lengths on a 64-bit system. Skimming the source, there are places where the author explicitly chooses to represent lengths with 32-bit types (e.g., schema.zig/readByteArray()). I bet you could massage the code to work with larger data without many issues.
- maccard 5y ago> I bet you could massage the code to work with larger data without many issues. I'm sure the author would appreciate a pull request if it's that straight forward.
- hansvm 5y agoSo it is that straightforward for a proof of concept (downgrade to an old version of Zig compatible with the project, patch an undefined variable bug the author introduced yesterday, s/32/64, add 4 bytes to main.zig->Header and its accesses). Doing so makes the program slower though which might be a non-starter for a performance focused project. Plus you'd need a little more work to properly handle large archives on 32-bit and smaller systems (at least those which support >4GB files).
- georgemcbay 5y agoThe answer is at the bottom of the page and all the offset/length data being uint32s. 2^32 = 4294967296 Not sure if this limitation is being enforced by upstream concerns, but this is why this code in particular is limited to that size.
- Jarred 5y agoI made this Happy to answer any questions or feedback
- ectopod 5y agoDon't use 32-bit file times! Change it quick while you have the chance.
- throwaway375 5y agoWhat do you think about Valve VPK?
- brandmeyer 5y agoIf you manage the index using a B-tree, then you can perform partial updates of the B-tree by appending the minimum number of pages needed to represent the changes. At that point, you can append N additional new files to the tail of the archive, and add a new index that re-uses some of the original index. Just an idea to check the "append" box. See also B-trees, Shadowing, and Clones https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf
- Jarred 5y agoGood idea, worried a little about impact to serialization/deserialization time though. Maybe could still store it as a flat array
- ksec 5y agoHow far are we from public beta or Bun 1.0? Not related to bun or hop is the use of Zig. I am wondering if you will do a write up on it someday.
- Jarred 5y agoSomething like two weeks before a public beta
- Const-me 5y ago
- perihelions 5y agoSpeaking of faster coreutils replacements, I highly recommend ripgrep (rg) and fd-find (fd), which are Rust-based, incompatible replacements for grep and find. I know I'm way behind the popularity curve* (they're already in debian-stable, for crying out loud); but for the benefit of anyone even more out of the loop than myself, do check these out. The speed increase is amazing: a significant quality-of-life improvement, essentially for free**. Particularly if you're on a modern NVME SSD and not utilizing it properly. (Incidentally, did you know the "t" in coreutil's "tar" stands for magnetic [t]ape?") * (The now-top reply to this comment says 'ag' is even superior to 'rg', and they're probably right, but I had no clue about it! I did say I'm ignorant!) **(Might have some cost if you're heavily invested in power-user syntax of the GNU or BSD versions, in which case incompatibility has a price). https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep https://github.com/sharkdp/fd https://github.com/sharkdp/fd
- chx 5y agoWhen I looked ag https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher was more featureful than ripgrep yet it's always ripgrep that is mentioned. :/
- rscnt 5y agoweird right? I don't know what ripgrep has, maybe it's just the name? the fact that you call `ag` but it's called `the_silver_searcher`?
- pdimitar 5y agoFrom my side, I knew about both `ripgrep` and `the_silver_searcher` but I will openly admit I've lost faith in C and C++'s abilities to protect against memory safety errors. Thus, Rust tools get a priority, at least for myself. There are probably other memory-safe languages but I haven't had the chance to give them a spin like I did with Rust. If I found out about them then I'll also prefer tools written in them if there's no Rust port and if the alternatives are C/C++ tools.
- zzo38computer 5y agoFile names not being normalized across platforms is sometimes beneficial. Ignoring symlinks is also sometimes beneficial. However, sometimes these are undesirable features. The same is true of solid compression, concatenatability, etc. Also, being able to effiently update an archive means that some other things may be lacked, so there is advantage and disadvantage of it. I dislike the command-line format of Hop; it seems to missing many features. Also, 64-bit timestamps and 64-bit data lengths (and offsets) would be helpful, as some other people mention; I agree with them, to improve it in this way. (You could use variable length numbers if you want to save space.) My own opinion for the general case is that I like to have concatenable format with separate compression (although this is not suitable for all applications). One way to allow additional features might be having an extensible set of fields, so you can include/exclude file modes, modifications times, numeric or named user IDs, IBM code pages, resource forks, cryptographic hashes, multi-volumes, etc. (I also designed a compression format with a optional key frame index; this way the same format supports both solid and non-solid compression, whichever way you want to do, and this can work independently from the archive format being used.) For making backups I use tar with a specific set of options (do not cross file systems, use numeric user IDs, etc); this is then piped to the program to compress it, stored in a separate partition, and then recorded on DVDs. For some simple uses I like the Hamster archive format. However, it also limits each individual file inside to 4 GB (although the entire archive is not limited in this way), and no metadata is possible. Still, for many applications, this simplicity is very helpful, and I sometimes use it. I wrote a program to deal with these files, and has a good number of options (which I have found useful) without being too complicated. Options that belong in external programs (such as compression) are not included, since you can use separate programs for that.
- Jarred 5y ago> I dislike the command-line format of Hop; it seems to missing many features. I agree 100%. I wrote most of it in like three hours; it's not a polished product. > My own opinion for the general case is that I like to have concatenable format with separate compression (although this is not suitable for all applications). One way to allow additional features might be having an extensible set of fields, so you can include/exclude file modes, modifications times, numeric or named user IDs, IBM code pages, resource forks, cryptographic hashes, multi-volumes, etc. (I also designed a compression format with a optional key frame index; this way the same format supports both solid and non-solid compression, whichever way you want to do, and this can work independently from the archive format being used.) I'm wary of slowing it down by adding lots of features. I think that, generally speaking, _more_ purpose-built binary formats should exist. Engineers do this all the time with YAML files and JSON, but why not binary files?
- euske 5y agoI used to use cdb a lot for random access of small key/value pairs (~100k entries, 10kb per entry). It's effectively a hashtable on disk. https://cr.yp.to/cdb.html https://cr.yp.to/cdb.html
- johnisgood 5y ago> No random limits: cdb can handle any database up to 4 gigabytes. I suppose this could easily be changed, right? I could not compile it though. I got this error: /usr/bin/ld: errno: TLS definition in /lib/x86_64-linux-gnu/libc.so.6 section .tbss mismatches non-TLS reference in cdb.a(cdb.o) /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: error adding symbols: bad value I never came across this before. Ideas as to why or how to fix? Nevermind, I fixed it. I added `#include <errno.h>` to `error.h`.