3 ms·
I wish there were a "zip" standard for storing a million small files in one package. NPM and other open source programs badly need this.
by nashashmi 4y ago
I wish there were a "zip" standard for storing a million small files in one package. NPM and other open source programs badly need this.
- Gigachad 4y agoIsn’t that just tar?
- ElectricalUnion 4y ago.jar/.apk (internally a zip archive) comes to mind. AppImage (internally is either a ISO 9660 Rock Ridge or a SquashFS filesystem), .deb (internally ar archive), .rpm (internally a cpio archive) are I think relevant examples too.
- Xamayon 4y agoI do exactly that for storing the result thumbnails for some of the dbs in my reverse image search engine (SauceNAO). Non compressed zip files allow quickly/easily seeking to and accessing component files without extraction. A few tens to hundreds of thousands per zip file works great. Millions would probably not be too different, but would use more resources/take more time when loading the zip file index.
- mthoms 4y agoInteresting. Have you ever considered SQLite file storage? I'm wondering how it would compare. https://www.sqlite.org/sqlar.html https://www.sqlite.org/sqlar.html
- Xamayon 4y agoHaven't looked into it, but it sounds like it would work similarly (with some nice benefits such as also being able to easily store other metadata/etc). Feasibility would depend on how quickly the indexes and such load, and the resource consumption associated with opening/closing dozens of them at a time 24/7. In my screwy case there are hundreds of thousands of zip files which are randomly accessed on the fly to grab one or two thumbnails at a time. The random access speed on unloaded files is critical, and for zip files it's extremely quick.
- eviks 4y agoExactly. Though yarn has this feature, and it works with zip files as mini-fs, so you don't need to unpack them to disk (generally)