9 ms·
Pack: A new container format for compressed files
- rodrigokumpera 3y agoZStandard is... standardized under rfc 8878 Plus there's no discussion against zstd itself and its container format.
- rustyconover 3y agoIf you're looking for a debate against ZStandard, its hard to argue against it. ZStandard is Pareto optimal. For the argument why, I really recommend this investigation. https://insanity.industries/post/pareto-optimal-compression/ https://insanity.industries/post/pareto-optimal-compression/
- bonki 3y agoThanks, superbly written and highly informative article!
- ramses0 3y agotldr: `cp * sqlite3://output.db` (basically?) Really seems to make sense! For another fun compression trick: https://github.com/mxmlnkn/ratarmount https://github.com/mxmlnkn/ratarmount
- xuhu 3y agoBetter, faster, stronger but I can't tell from the homepage what's different about it, except that it is based on SQLite and Zstd.
- OttoCoddo 3y agoYou may like to read https://pack.ac/note/pack https://pack.ac/note/pack and test it for yourself.
- kevmo314 3y agoWow, Pascal! Haven't seen a project in Pascal in a while. https://github.com/PackOrganization/Pack https://github.com/PackOrganization/Pack
- nkozyra 3y agoYeah, I'll wait for the ALGOL 68 port.
- jksmith 3y agohttps://en.wikipedia.org/wiki/Argumentum_ad_populum https://en.wikipedia.org/wiki/Argumentum_ad_populum can be a mistake of the youth. Just sayin.
- nkozyra 3y agoPascal was my first programming language, but I appreciate the link ;)
- OttoCoddo 3y agoThat is the best joke I've heard all day. Thank you for the laugh :)
- p0w3n3d 3y agoLast time is did some Pascal was 2006/7. I think I never saw production-grade code myself. I wonder if this line is an array in-situ? Split(APath, [sfodpoWithoutPathDelimiter, sfodpoWithoutExtension], P, N)
- mananaysiempre 3y agoNo, it’s a set (bitmask) constant[1]. [1] https://www.freepascal.org/docs-html/current/ref/refse83.html https://www.freepascal.org/docs-html/current/ref/refse83.htm...
- jksmith 3y agoBut yeah FPC supports ref counted dynamic arrays with "+" operator.
- bborud 3y agoPascal!? My monocle nearly fell out.
- mmastrac 3y agoSqlite3 is universal, but now your spec is entirely reliant on Sqlite3's format and all the quirks required to support that. If you actually care about the future, spec out your own database format and use that instead. It could even be mostly a copy of Sqlite3, but at least it would be part of the spec itself.
- kevmo314 3y agoOn the other hand, by using Sqlite one can reimplement this format in another language with very little effort.
- KerrAvon 3y agoIt requires the sqlite3 library bindings, which might be a lot of effort.
- 0cf8612b2e1e 3y agoIs there a mainstream language which does not have SQLite bindings?
- remram 3y agoThat's not the usual meaning of "reimplement".
- lordmauve 3y ago
- jlhawn 3y agoWhen I read the title, I thought it was a new operating system-level containerization image format for filesystem layers and runtime config. But it looks like "container format" is a more general term for a collection of multiple files or streams into one. https://en.wikipedia.org/wiki/Container_format https://en.wikipedia.org/wiki/Container_format TIL. OS containers could use an update too, though. They're often so big and tend to use multiple tar.gz files.
- theamk 3y ago> Pack format is based on the universal and arguably one of the most safe and stable file formats, SQLite3, and compression is based on Zstandard, the leading standard algorithm in the field. yeah, no thanks. SQlite3 automatically means: - Single implementation (yes, it's a nice one but still a single dependency) - No way to write directly to pipe (SQlite requires real on-disk file) - No standard way to read without getting the whole file first - No guarantees in number of disk seeks required to open the file (relevant for NFS, sshfs or any other remote filesystem use) - The archive file might be changed just by opening in read-only mode - Damaged file recovery is very hard - Writing is explicitly not protected against several common scenarios, like backup being taken in the middle of file write - No parallel reads from multiple threads Look, sqlite3 is great for it's designed purpose (embedded database). But trying to apply for other purposes is often a bad idea.
- bno1 3y agoI found squashfs to be a great archive format. It preserves Linux file ownership and permissions, you can extract individual files without parsing the entire archive like tar and it's mountable. It's also openable in 7zip. I wonder how pack compares to it, but its home page and github don't tell much.
- benibela 3y agoBut it is read-only? I was trying to change a single file in squashfs container recently and could not find a way to do that.
- bonki 3y agoI second this.
- deleted 3y ago[deleted]
- Aachen 3y agoThat's exactly what I'd like to avoid. I want to transfer a group of files (either to myself, friends, or website visitors), not make assumptions about the target system's permission set. For copies of my own data where permissions are relevant, I've got a restic backup Wake me up if a simple standard comes to pass that neither has user/group ID, mode fields, nor bit-packed two-second-precision timestamps or similar silliness. Perhaps an executable bit for systems that insist on such things for being able to use the download in the intended way (I self-made this before: a simple length-prefixed concatenation of filename and contents fields. The problem is that people would have to download an unpacker. That's not broadly useful unless it is, as in that one case, a software distribution which they're going to run anyway)
- lmz 3y ago> Wake me up if a simple standard comes to pass that neither has user/group ID, mode fields, nor bit-packed two-second-precision timestamps or similar silliness. Perhaps an executable bit for systems that insist on such things for being able to use the download in the intended way Other than having timestamps isn't this a ZIP file? No user id, no x bit, widely available implementations... Not very simple though I guess.
- Kwpolska 3y agoIf I need to compress stuff, it’s either to move a folder around (to places which may not have a niche compression tool, so ZIP wins), or to archive something long-term, where it can take a while to compress. I don’t see the advantages of this, since the compression output size seems quite mediocre even if it’s supposedly fast (compared to what implementations of the other formats?)
- xcdzvyn 3y agoWith all due respect, I find it hard to believe the author stumbled upon a trivial method of improving tarballing performance by several orders of magnitude that nobody else had considered before. If I understand correctly, they're suggesting Pack, which both archives and compresses, is 30x faster than creating a plain tar archive. That just sounds like you used multithreading and tar didn't. Either way, it'd be nice to see [a version of] Pack support plain archival, rather than being forced to tack on Zstd.
- TylerE 3y agoThat’s more because plain tar is actually a really dumb way of handling files that aren’t going to tape. Being better than that is not a hard bar.
- cogman10 3y agoThe tar file format is REALLY bad. It's pretty much impossible to thread because it's just doing metadata then content and repeatably concatenating. IE /foo.txt 21 This is the foo file /bar.txt 21 This is the bar file That makes it super hard to deal with as you essentially need to navigate the entire tar file before you can list the directories in a tar file. To add a file you have to wait for the previous file to be added. Using something like sqlite solves this particular problem because you can have a table with file names and a table with file contents that can both be inserted into in parallel (though that will mean the contents aren't guaranteed to be contiguous.) Since SQLite is just a btree it's easy (well, known) how to concurrently modify the contents of the tree.
- TylerE 3y agoOr just what zip and every other format does an skits put all the metadata at the beginning - enough to list all files, and extract any single one efficiently
- nullindividual 3y agoTapes don't (? certainly didn't) operate this way. You need to read the entire tape to list the contents. Since tar is a Tape ARchive, the way tar operates makes sense (as it was designed for both output to file and device, i.e. tape).
- crq-yml 3y agoThe web site behaves strangely on mobile and folds the text as I try to scroll around.
- 0x073 3y agoYes at least on mobile absolute madness.
- OttoCoddo 3y agoSorry. Site is very new, and custom made, and needs to be worked on mobile.
- indrora 3y agoMy suggestion? If you’re not sure, grab a very basic template from either Hugo, MkDocs, or GitHub pages. They’re all pretty well tested and hard to break.
- smartmic 3y agoThe whole thing makes sense to me and I can't see any major points of criticism in the design rationale. Some thoughts: * There is already a "native" Sqlite3 container solution called Sqlar [0]. * Sqlite3 itself is certainly suitable as a base and I wouldn't worry about its future at all. * Pascal is also an interesting choice, it is not the hippest language nor a new kid on the block, but offers its own advantages as being "boring" and "normal". I am thinking especially of the Lindy effect [1]. All in all a nice surprise and I am curious to see the future of Pack. After all, it can only succeed if it gets a stable, critical mass of supporters, both from the user and maintainer spectrum. [0]: https://sqlite.org/sqlar/doc/trunk/README.md https://sqlite.org/sqlar/doc/trunk/README.md [1]: https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect
- sgammon 3y agoRelated note: Lindy's closed in 2018. https://en.wikipedia.org/wiki/Lindy%27s https://en.wikipedia.org/wiki/Lindy%27s
- epcoa 3y agoAnd that wasn’t even the original Lindy’s but an unrelated/“unauthorized” reboot since the trademark was declared abandoned. The original closed in 1969 (for which the law was named as it predates that): from 1964 https://web.archive.org/web/20210619015733/https://www.gwern.net/docs/statistics/1964-goldman.pdf https://web.archive.org/web/20210619015733/https://www.gwern...
- OttoCoddo 3y agoThank you so much for the kind words, refreshing. Here is the latest sqlar result on Linux source code on the same test machine in warm state: sqlar: 268 MB, 30.5 s Pack: 194 MB, 1.3 s Very good result compared to tar.gz. And much better than ZIP, considering sqlar gives random access like ZIP and unlike tar.gz. I considered sqlar as a proof of concept, and it inspired me to create Pack as a full solution. I always agreed with the great drh (creator of SQLite) points about SQLite as a file format, and Pack is a try to demonstrate that. I made Pack to give people a better life (at least behind their desks), and as you do, I hope people get to use it and find it useful.
- SyrupThinker 3y agoInteresting, I've recently spent an unhealthy amount of time researching archival formats to build the same setup of using SQLite with ZStd. My use case is extremely redundant data (specific website dumps + logs) that I want decently quick random access into, and I was unhappy with either the access speed, quality/usability or even existence of libraries for several formats. Glancing over the code this seems to use the following setup: - Aggregate files - Chunk into blocks - Compress blocks of fixed size - Store file to chunk and chunk to block associations What I did not see is a deduplication step for the chunks, or an attempt to group files (and by extend, blocks) by similarity in an attempt improve compression. But I might have just missed that due to lack of familiarity with Pascal. For anyone interested in this strategy, take a look at ZPAQ [1] by Matt Mahoney, you might know him from the Hutter Prize competition [2] / Large Text Compression Benchmark. It takes 14th place with tuned parameters. There's also a maintained fork called zpaqfranz, but I ran into some issues like incorrect disk size estimates with it. For me the code was also sometimes hard to read due to being a mix of English and Italian. So your mileage may vary. [1]: http://mattmahoney.net/dc/zpaq.html http://mattmahoney.net/dc/zpaq.html [2]: http://prize.hutter1.net http://prize.hutter1.net [3]: https://github.com/fcorbelli/zpaqfranz https://github.com/fcorbelli/zpaqfranz
- juitpykyk 3y agoFor your use case you might want to look at RocksDB. It supports zstandard compression, random access, and it's very robust.
- OttoCoddo 3y agoThank you for the detail check. I should thank the syrup too :) I'm happy to see a fellow enthusiast. Your deduction is on point. And also, Pack is smart; it skips non-compressible files like MP3 [1], so you do not need to choose the "Store" option to have a faster option, and it speedup decompression too. Pack is the first to achieve this, being faster than Store options. Yes, it was a surprise to me too. ZPAQ is great, and I study the Hutter Prize competition. Pack is on another chart, which is why I proposed CompressedSpeed [2]. The speed of getting to compression needs to be accounted for. You can store anything on an atom if you try hard enough, but hard work takes time. Deduplication step may get added, but in Hard Press [3]. I am curious to see the results of Pack on your data. You can find me here or o at pack.ac. [1] It is based on content rather than extension; any data that is determined not to be worthy of compression, will be stored as is. And as a file can get chucked, some parts can get compressed and some cannot. Imagine that part of the subtitle in a MKV file can get compressed, and the Video part gets skipped. Although these features will get more updates over time, if they don't cost time,. Pack focus is being seamless and not the most compressed; there are already great works in the field, such as the noted ZPAQ. [2] CompressedSpeed = (InputSize / OutputSize) * (InputSize / Speed). Materialized compression speed. [3] You can choose --press=hard to ask for better compression. Even with Hard Press, Pack does not try to eat your hardware just to get a little more; it goes the optimized way I described.
- ericyd 3y agoWebsite is quite annoying to use on mobile. Scrolling behaviors get interpreted as taps which close the container you're reading.
- throwaway67743 3y agoAlso if you dare try to naturally scroll up after opening a container it's interpreted as a refresh as it redraws. Might be an awesome format but web design fail negates it entirely.
- megraf 3y agoYes, it's horrendous - I thought I was intoxicated
- cpressland 3y agoHad the same issue, plus I think one of my Safari extensions was hiding a cookie banner so half the page was just “dark”. A more mobile friendly view is absolutely needed.
- croemer 3y agoNah, there's no cookie banner. It's just a strange/innovative design.
- deleted 3y ago[deleted]
- conception 3y agoIs anyone using this or sqllite archives for anything at scale? They always seemed like a good solution for certain scientific outputs. But data integrity obviously is a concern.
- Kalanos 3y agois this as widely compatible as 7zip?
- chatmasta 3y agoWhat's the best way to store a bunch of .zip files that all share some data? Assume that I can't decompress or alter the zip files in any way. Basically I want a shared encoding dictionary. Is there an easy solution for this? The use case is maintaining an archive of .crx files.
- sph 3y agoNot sure if I understand the question. Having a shared encoding dictionary is called compression. If you want one across many zip files that you cannot modify, then compress them into another archive.
- chatmasta 3y agoYes, I have a bunch of zips I cannot modify. If I could uncompress them all, and zip them into one big archive, then it would be much smaller than if I put all the compressed zips into one big archive. This is because there's a lot more shared strings between uncompressed files across different zips, than there are shared strings between the compressed zips. (At least... this is my assumption based on how I understand the formats to work. I do need to measure and verify this.) So basically I'm wondering if there is some way to tell the "outer" zip to reuse the encoding dictionaries of each smaller zip, or to somehow intelligently merge their encoding dictionaries rather than treating each inner zip like an opaque blob.
- teraflop 3y agoNo, the ZIP format compresses each embedded file separately, so it doesn't matter how much commonality there is between different files in the same archive.
- chatmasta 3y agoWow, really? I always assumed that if I zipped a directory, it would use a shared dictionary to compress all the files. But I guess what you say makes sense, because otherwise it wouldn't be possible to extract just one file. Are there compression formats that do share an encoding dictionary across multiple files? I guess tar + gzip might do that?
- Aachen 3y agoComplaining that all existing tools are old, but I'm looking at the documentation and the what immediately catches my eye is that it doesn't use any modern convention I've gotten used to? Overwrite with "-w". I've never seen a tool not use "-f" Not reserving "-h" for help text is also an interesting choice. Makes me think of the mantra "be conservative in what you send out but liberal in what you accept". Per that philosophy, both "--help" and "-h" should be accepted because neither gained a decisive majority in usage and so people might try either. It's not like you'd know what to use because it hasn't told you yet Forcing use of a long option "--press=N" (for the zstd level setting) is also new/unique terminology for what is usually "-N" (like "-1" to "-9") (Basic) drop-in compatibility with every other tool from gzip to zstd would also have been nice, but archiving and compression are different things and everything from zip to 7z to tar works in unique ways so this makes enough sense I guess. Still, could have been useful It's still better than tar or ps, so if it catches on that's still a step forwards in terms of command line standards
- deleted 3y ago[deleted]
- drpixie 3y agoHmmm. I went to the doco hoping for something about the file format :( No doco for me. I guess that would be too old fashioned. It seems to be "read the code" or nothing - which is fine until they update the code... It's great (probably should be mandatory) to have a reference program, but if they're promoting it as a container format, something along the lines of an RFC would be helpful.
- OttoCoddo 3y agoMore documentation will be published soon. For now and about CLI: https://pack.ac/cli-documentation https://pack.ac/cli-documentation
- OttoCoddo 3y agoHey, The choice of parameters was solely done to be clear, and not what people used to. -f meaning force is not clear; -w meaning overwrite, seemed like a better logical choice, to me. Nice point on -h. Yes, I did not want to go crazy. After all, almost all (CLI) people use pack as `pack ./test/`. Options are for advanced people like you. Most people will use the OS integration that will be published later on. --press=hard is the only option there is. There may be more, but with Pack you do not need to choose a level (like 1..9 with ZIP). Just let Pack do its thing, and you will be happy. Hard Press is there for people who want to pack once and unpack many times (like publishing), and it is worth spending extra time on it. Even then, Pack goes the sane way and does not eat your computer just for a kilobyte or two.
- jksmith 3y agoHell yeah some fpc stuff showing its moves. Devs even put together an lpk to load up -bravo! Look for more of this stuff in the future as companies look for alternatives to corporate commodity programming and dbs tethered to major cloud resources. I have a major FPC effort going on right now that I hope to be able to offer on four platforms: browser, win, linux, and mac. I wrote a similar general purpose pack way back in the early 90's in TopSpeed Modula-2 and run from the command line. Needed to span multiple disks and self launch. Algorithm was fast, but not nearly the same compression ratios. Wore out Mark Nelson's classic, "The Data Compression Book" along the way.
- OttoCoddo 3y agoHell yeah indeed! You can find me on the forum too if you liked to talk Pascal: https://forum.lazarus.freepascal.org/index.php/topic,66281.0.html https://forum.lazarus.freepascal.org/index.php/topic,66281.0...
- croemer 3y agoWho is behind this? It's a new Github org, the committer (https://github.com/OttoCoddo https://github.com/OttoCoddo) has a totally private Github profile. There's no name in "Legal". Sure, one can be anonymous, but I won't download it, don't trust it.
- Retr0id 3y agoThey also mention depending on https://github.com/SCLOrganization https://github.com/SCLOrganization, which appears to have single similarly anonymous maintainer. In fact, SCL has exactly two libraries, SQLite and Zstandard, so presumably it's the same developer https://github.com/SCLOrganization/Libraries https://github.com/SCLOrganization/Libraries
- OttoCoddo 3y agoMe. It is the point: if you trust a project based on "who" made it, my friend, that is the start of the big problem we are facing in this current situation of tech. Just look at the code, build it yourself, and check the license. Pack is made to be a private option; future locking end encryption options will solidify that. Trusting the author is not the correct way to verify the security and safety of such a tool.
- sgammon 3y agoI remember someone had made a package manager / index tool for SQLite extensions. But I haven't ever been able to find it again
- toomuchtodo 3y agosqlean or sqlpkg perhaps? https://github.com/nalgeon/sqlean https://github.com/nalgeon/sqlean https://sqlpkg.org/ https://sqlpkg.org/
- sgammon 3y agoI think it was sqlpkg! Thank you. That was bothering me so much!
- Retr0id 3y agosqlite is great as a "file format" for a particular application, but I think it's a bad interchange format. As mediocre as zip and tar are, you can cobble together read/write support without even needing a library. With sqlite, your only real option is to bundle sqlite itself, and while it's relatively lightweight, it's far from trivial. zip has support for zstd, and if you wanted to make it go faster, you could embed some index metadata. I can't see any specs for their format, not even a description of the sqlite tables.
- Retr0id 3y agoAfter overwriting the "Pack" magic bytes back to the SQLite default values, I was able to open it and see the following tables CREATE TABLE Content(ID INTEGER PRIMARY KEY, Value BLOB); CREATE TABLE Item(ID INTEGER PRIMARY KEY, Parent INTEGER, Kind INTEGER, Name TEXT); CREATE TABLE ItemContent(ID INTEGER PRIMARY KEY, Item INTEGER, ItemPosition INTEGER, Content INTEGER, ContentPosition INTEGER, Size INTEGER); According to the `.indexes` directive, there are... no indexes. What's the point of sqlite if you're not going to index things? All the data is stored in one big blob (the "Value" column of the "Content" table), with the metadata storing offsets into it. It looks like there's still the possibility of things being split over multiple blobs (to circumvent the 2GB blob size limit)
- Retr0id 3y agoI've reverse engineered the format and written up my findings here: https://github.com/DavidBuchanan314/pack-analysis https://github.com/DavidBuchanan314/pack-analysis Summary: - Custom sqlite magic bytes makes the format incompatible with all existing sqlite tooling. - No support for file metadata. - There's no version field (afaict), making future format improvements difficult. Edit: A previous version of this comment had a much longer list of complaints, but after taking a closer look, I retract them. I was looking at the MediaKit.pack file as an example, which, due to being relatively small, packed all its files into a single BLOB. I was under the mistaken impression that the same approach was taken for larger files, but after some further testing I see that they're split up into ~8MB chunks. Though, if you have lots of small files (say, a couple of kilobytes each) then random access performance could suffer.
- deleted 3y ago[deleted]
- paulddraper 3y agoHow does tar take 4.7s in the benchmark, but Pack takes 1.3s (and 1/7th the size)? That seems fishy...
- t43562 3y agoI had to make an archiver once (commercial) so I did think about it a bit. I am not sure pack would solve anything for me. It obviously solves the authors usecases but tar has some tricks which I don't want to lose; * Able to write to a pipe/socket - lets you not waste space or time by writing to disc something that you intend to transmit over a pipe or TCP socket anyhow. It's almost a "virtual" archive and it should be possible to make one that is far too big to fit into memory - because as you send each bit of it you deallocate that memory. At the receiver each bit can be written to disc or extracted and then that memory is reused for the next bit - so the archive never fully "exists" but it does the job of serialising some data. An example could be piping the output of tar to an ssh command which untars it on a remote machine. * Metadata has to be with the file data - not stuck at the end of the file - because you need to be able to start work without waiting till the file is fully received through your pipe. You don't want to be forced to have space to store the archive and the extracted files (may be a huge archive). * Choice of compression - lzop is super fast such that using it can sometimes give slightly better performance than writing uncompressed data. OTOH that might not be your concern and XZ might suit you by compressing much more thoroughly. Either way it's very nice to have compression that works across multiple files - which is especially helpful when compressing a lot of small files such as source code. * Ability to encapsulate - should be able to put the packed data into any imaginable container like an encryption or data transmission protocol without insisting that the entire archive has to be fully read before members can start to be extracted/processed. This is essentially the same as the pipe/socket requirement. I'm not saying that these things matter to everyone - I have just found them incredibly useful in a few critical situations. The world of ZIP users on Windows seems to be sort of blind to them - thinking firmly in that box.
- masklinn 3y ago> The world of ZIP users on Windows seems to be sort of blind to them - thinking firmly in that box. Seems to me like this is you being blind to certain use cases, and so stuck in your streaming oriented box that you can not conceive of other use cases where a streaming format is actively detrimental.
- t43562 3y ago
- rurban 3y agoUnfortunate name: dnf info pack Summary : Convert code into runnable images URL : https://github.com/buildpacks/pack License : Apache-2.0 and BSD-2-Clause and BSD-3-Clause and ISC and MIT Description : pack is a CLI implementation of the Platform Interface Specification : for Cloud Native Buildpacks.
- SillyUsername 3y agoHow does this compare to PAQ https://en.m.wikipedia.org/wiki/PAQ https://en.m.wikipedia.org/wiki/PAQ ?
- OttoCoddo 3y agoAs far as I know, PAQ strives to give the best compression for archival purposes. Pack, on the other hand, tries to give the best compression, while keeping speed as instantaneous as possible. You should try them for yourself.
- hexo 3y agoQuite a hilarious joke, hope no one takes this seriously
- OttoCoddo 3y agoHello to all. I am the author, and I just saw this post and am happy to see this exciting discussion. Let me try to show my respect for it and answer as well as possible.
- jlundberg 3y agoWelcome Otto! I know many people including myself are curious on why you wrote this in Pascal? Also, what are the main reasons Pack is faster than the tools you compare Pack with?
- OttoCoddo 3y agoThank you! About Pascal: It makes me happy. It looks clean and pseudocode-like. It helps readers from around the world with different languages understand. I am happy that Pack made people curious about this old but great goodie. Speed, Here are some reasons: - Pack does all the steps of pack or unpack (read, check, compress/decompress and write) together and this weaving helped achieve this speed and random access. It is by far the fastest speed I get to see reading or writing random files from file systems, as fast or faster than asynchronous read operations or OVERLAPPED on Windows. To a point, it is limited to file system. For example, on NTFS, Pack can pack Linux code base in around 1.3 s; similar is done on ext4 in 0.96 s. - It is based on a heavily optimized code base, standard library, and the FreePascal compiler, which produces great binary. - Multi core design: even mobiles have multi-core CPUs these days. Choosing threads based on the content and machine, it does not eat your machine. - Speed-configured SQLite. SQLite is much faster than most people think it is. - Configured the already rapid Zstandard. In summary, standing on the shoulders of giants while trying hard to improve reliability, speed and user experience is a sign of respect for them.
- abcd_f 3y agoThe website is completely unusable on iPhone and on iPad. It doesn't scroll, jerks back in place, parts are blank and empty... looks really strange.
- OttoCoddo 3y ago
- evnix 3y agoAny recommendation for a format that supports editing and deleting files without rewriting the whole archive
- nojvek 3y agoSite is broken on mobile. See logo and black screen. Reader mode doesn’t work.
- 2color 3y agobeautiful website and nice overall aesthetic.
- OttoCoddo 3y agoThank you very much! I guess you are friend with colors ;)
- out_of_protocol 3y agoHow about pure zstd (or tar.zstd) vs pack vs patched 7z+zstd benchmark? Measure container overhead, in both speed and bytes
- OttoCoddo 3y agoI answered this question here: https://news.ycombinator.com/item?id=39801083 https://news.ycombinator.com/item?id=39801083 If that is not enough, let me know. 7-Zip with the ZSTD patch is good too, but Pack is much faster at handling many files. Testing packing the Linux code base (81K files and 1.25 GB) on Windows with NTFS: 7-Zip + Patched with ZSTD (-m0=Zstd): 6.453 s, 194.9 MB (Creating the header takes too much time) Pack: 1.3 s, 194.5 MB
- out_of_protocol 3y agoThanks for the update! Somehow 7z container overhead is +0.4MB AND is slower by a lot? Huh. Great numbers, need more exposure for pack format. Also, i suggest to add section to the website that shows these numbers. 1) regular archive formats vs pack 2) various containers with the same zstd inside.
- OttoCoddo 3y agoThank you! Exposure comes from enthusiasts like you. I did not want to focus the point on speed, or say, "Look, others are bad". They are great; my point was, "Look what we can do if we update our design and code". Pack value comes from user experience, and speed is being one. I was not following the best speed or compression; I wanted an instantaneous feeling for most files. I wanted a better API, an easier CLI, improved OS integration (soon), and more safety and reliability. Tech people (including me) care so much about speed I am happy about the results, but Pack offers much more that I like others to see.
- deleted 3y ago[deleted]
- raggi 3y agoI went to do some testing in a sandbox system (as the compilation strategy is unclear, missing builds for some of the artifacts). I was able to initially construct an archive of the linux tree (that failed in decompression), but subsequently went to rebuild it and the tool is repeatedly producing this output even in an otherwise cleaned up environment: Runtime error 203 at $0000000100009B72 $0000000100009B72 $00000001000225B6 $0000000100023153 $000000010002319A Runtime error 203 at $0000000100009B72 $0000000100009B72 $00000001000225B6 $0000000100023153 $000000010002319A Runtime error 203 at $0000000100009B72 $0000000100009B72 $00000001000225B6 $0000000100023153 $000000010002319A Runtime error 203 at $0000000100009B72 $0000000100009B72 $00000001000225B6 $0000000100023153 $000000010002319A Runtime error 203 at $0000000100009B72 $0000000100009B72 $00000001000225B6 $0000000100023153 $000000010002319A The first output did work, sadly I deleted it. It was over 400mb, so approximately double the size of a zip or a tar.zst of the same files. zstd natively compresses a tar of this set to 209mb in 800ms in multi-threaded mode, or 3.5s in single threaded mode. I suspect that sqlite is being held incorrectly (access from multiple threads, with multi-threading disabled), and the vfs lock forwarding is broken on Windows.
- OttoCoddo 3y agoDid you compile it for yourself? Any problem or steps you used, I will be happy to hear, o at pack.ac or GitHub, as it is hard to follow the building here. I should prepare more documents on how to build it. I suspect that there is a problem with the custom build. Error and speed issues are not something you see in the official build.
- raggi 3y agoThat was the binary download from the website. You have a build.sh for the Linux binary artifacts, but no equivalent for the windows artifacts so i did not bother preparing a windows build
- OttoCoddo 3y agoPack binary? Can you tell what machine and what steps? Build.sh can be used for Windows too, using MSYS2 UCRT64.
- deleted 3y ago[deleted]
- 404mm 3y agoIt’s written in Pascal.. Why not something like Rust or at least C++?
- OttoCoddo 3y agoIt makes me happy. It looks clean and pseudocode-like. It helps readers from around the world with different languages understand.
- SolitudeSF 3y agono build instructions, binary artifacts in repo. yuck.
- OttoCoddo 3y agoEasy to build using this document: https://pack.ac/source https://pack.ac/source Each binary has its own build script that you can use for yourself. Binaries are used for static builds and to ease future needs. https://github.com/PackOrganization/Pack/blob/main/Libraries/Zstandard/Build/Build.sh https://github.com/PackOrganization/Pack/blob/main/Libraries... I know you don't have a duty to look around for your answer, but you too don't have a duty to say yuck to a project that has been done with a lot of effort. I am ok with your comment, but maybe go easy on the next project.