6 ms·

Squashfs turning 20, Squashfs tools 4.5 released

69 points by st_goliath 5y ago

usr1106 5y ago
Nice that squashfs is alive. I tried to use it in a project about 4-5 years ago. It turned out that read performance was much worse than from ext4. Ext4 was able to submit block reads in big units. Squashfs submitted many times more tiny requests to the block layer. The git log looked like the code had been abandoned. Has that changed, too? The release mentioned here is user space, not kernel.
- pabs3 5y ago
  The Linux kernel squashfs code had 17 commits in 2021 & 2020: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/fs/squashfs https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - bicolao 5y ago
    And this one in particular says decreases boot time by ~40% https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/squashfs?id=93e72b3c612adcaca13d874fcc86c53e6c8da541 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- derefr 5y ago
  I’ve always been curious what the most optimal “network-portable mountable filesystem image” format for Linux would be (for e.g. being a zero-install auto-fetched-from-the-network “mount and run” format for applications ala Flatpak/Snap; or being a NeXT-alike distribution container for “bundles” — where, as in macOS, it’s just as viable to open the app/document bundle [read-only] from the image, as it is to copy it out of the image.) Such a format would have to balance the considerations of download time, storage size at rest, mount time (incl. any integrity checking), and access-time overhead when mounted. It would especially have to worry about per-file-open overhead, given that one of the primary use-cases would be rsync-ing the file tree out of the image onto your disk. It sounds like you’re saying that a SquashFS filesystem image would actually be bad for this use-case, vs. just downloading a regular (sparse) loopback image of an ext4 filesystem over a compressed transport. Which is kind of surprising to me! I would think that the use-cases stated above would have been one of the design goals. I’d also be curious about the relative merits of the “pure userspace” option — i.e. rather than mounting a disk-image and then accessing the files inside it through regular VFS inodes, instead having your zero-install daemon or whatever just use regular tarballs, by linking in a library like libarchive and using it to get internal virtual handles on the files. (Lots of games seem to choose this option, so it must have pretty good perf.)
  - selfhoster11 5y ago
    I had a similar problem. Large archive files, quick random access was paramount. I considered SquashFS, but rejected it due to poor tooling on Windows, and I wanted my thing to be cross-platform. The format also seemed like something I'd need to spend some time understanding. Surprisingly, ZIP came to my rescue. It's surprisingly flexible, and includes support for Unix attributes and modern compression. In comparison to tar.xz, it was strictly superior in every way. ISO is also a good format with broad support, with many platforms now supporting mounting the contents using the built-in OS features. The only downside I could find is lack of compression, but there are supposedly format extensions that can help with that.
hawski 5y ago
I see some features from squashfs-tools-ng like cpio style list of entries. Are those projects being merged or are they a totally separate effort?
- st_goliath 5y ago
  Unlikely. Squashfs-tools-ng is not a fork but written from scratch after reverse engineering the format. I started it in early 2019 after being annoyed with various shortcomings in squashfs-tools and after trying to make sense out of the spaghetti source code to not only fix problems I ran into, but also add features I needed for a particular project. I saw the complete lack of activity on the official website and official git tree, from which I concluded it was pretty much dead. Of course I wanted to ask on the mailing list first (also pretty dead), but then I saw a thread from early 2018 talking about a release. Since the thread got a response that I interpreted as rather angry[1] and the release didn't happen a year later either, I concluded that the project is pretty much dead and decided to publish my own efforts and for lack of a better name simply named it "*-ng". Other people seemed to have drawn similar conclusions[2][3]. We all later found out that the squashfs-tools git tree was simply moved to GitHub (without updating the kernel docs or "official" website to indicate this). But even there, activity fixing bugs only picked up after I already made 3 or 4 releases and announced them on the squashfs mailing list. At which point I finally got a response in the form of a few somewhat unfriendly words[5][6][7] (IMO to some extent understandable, but still...). But after that, squashfs-tools version 4.4 fixed a bunch of issues that I publicly complained about and this release got apparently quite busy copying features that I found useful (and set squashfs-tools-ng apart, e.g. converting tarballs directly). Given the distinct source trees and, lets call it "competitive nature", I'd say a merge is unlikely and I will continue maintaining my package. I wrote it to solve some specific problems I had and it IMO does that job quite well. It's nice if other people find it useful and I'm also proud to have ended up indirectly contributing to the official squashfs-tools development by providing new motivation :-). [1] https://sourceforge.net/p/squashfs/mailman/message/36298345/ https://sourceforge.net/p/squashfs/mailman/message/36298345/ [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965 [3] https://news.ycombinator.com/item?id=20518019 https://news.ycombinator.com/item?id=20518019 [4] https://lkml.org/lkml/2019/8/1/1154 https://lkml.org/lkml/2019/8/1/1154 [5] https://github.com/AgentD/squashfs-tools-ng/issues/10 https://github.com/AgentD/squashfs-tools-ng/issues/10 [6] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#29 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#29 [7] https://lkml.org/lkml/2019/8/1/565 https://lkml.org/lkml/2019/8/1/565

69 points by st_goliath 5y ago

Nice that squashfs is alive. I tried to use it in a project about 4-5 years ago. It turned out that read performance was much worse than from ext4. Ext4 was able to submit block reads in big units. Squashfs submitted many times more tiny requests to the block layer. The git log looked like the code had been abandoned. Has that changed, too? The release mentioned here is user space, not kernel.

pabs3 5y ago
The Linux kernel squashfs code had 17 commits in 2021 & 2020: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/fs/squashfs https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- bicolao 5y ago
  And this one in particular says decreases boot time by ~40% https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/squashfs?id=93e72b3c612adcaca13d874fcc86c53e6c8da541 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
derefr 5y ago
I’ve always been curious what the most optimal “network-portable mountable filesystem image” format for Linux would be (for e.g. being a zero-install auto-fetched-from-the-network “mount and run” format for applications ala Flatpak/Snap; or being a NeXT-alike distribution container for “bundles” — where, as in macOS, it’s just as viable to open the app/document bundle [read-only] from the image, as it is to copy it out of the image.) Such a format would have to balance the considerations of download time, storage size at rest, mount time (incl. any integrity checking), and access-time overhead when mounted. It would especially have to worry about per-file-open overhead, given that one of the primary use-cases would be rsync-ing the file tree out of the image onto your disk. It sounds like you’re saying that a SquashFS filesystem image would actually be bad for this use-case, vs. just downloading a regular (sparse) loopback image of an ext4 filesystem over a compressed transport. Which is kind of surprising to me! I would think that the use-cases stated above would have been one of the design goals. I’d also be curious about the relative merits of the “pure userspace” option — i.e. rather than mounting a disk-image and then accessing the files inside it through regular VFS inodes, instead having your zero-install daemon or whatever just use regular tarballs, by linking in a library like libarchive and using it to get internal virtual handles on the files. (Lots of games seem to choose this option, so it must have pretty good perf.)
- selfhoster11 5y ago
  I had a similar problem. Large archive files, quick random access was paramount. I considered SquashFS, but rejected it due to poor tooling on Windows, and I wanted my thing to be cross-platform. The format also seemed like something I'd need to spend some time understanding. Surprisingly, ZIP came to my rescue. It's surprisingly flexible, and includes support for Unix attributes and modern compression. In comparison to tar.xz, it was strictly superior in every way. ISO is also a good format with broad support, with many platforms now supporting mounting the contents using the built-in OS features. The only downside I could find is lack of compression, but there are supposedly format extensions that can help with that.

hawski 5y ago

I see some features from squashfs-tools-ng like cpio style list of entries. Are those projects being merged or are they a totally separate effort?

st_goliath 5y ago
Unlikely. Squashfs-tools-ng is not a fork but written from scratch after reverse engineering the format. I started it in early 2019 after being annoyed with various shortcomings in squashfs-tools and after trying to make sense out of the spaghetti source code to not only fix problems I ran into, but also add features I needed for a particular project. I saw the complete lack of activity on the official website and official git tree, from which I concluded it was pretty much dead. Of course I wanted to ask on the mailing list first (also pretty dead), but then I saw a thread from early 2018 talking about a release. Since the thread got a response that I interpreted as rather angry[1] and the release didn't happen a year later either, I concluded that the project is pretty much dead and decided to publish my own efforts and for lack of a better name simply named it "*-ng". Other people seemed to have drawn similar conclusions[2][3]. We all later found out that the squashfs-tools git tree was simply moved to GitHub (without updating the kernel docs or "official" website to indicate this). But even there, activity fixing bugs only picked up after I already made 3 or 4 releases and announced them on the squashfs mailing list. At which point I finally got a response in the form of a few somewhat unfriendly words[5][6][7] (IMO to some extent understandable, but still...). But after that, squashfs-tools version 4.4 fixed a bunch of issues that I publicly complained about and this release got apparently quite busy copying features that I found useful (and set squashfs-tools-ng apart, e.g. converting tarballs directly). Given the distinct source trees and, lets call it "competitive nature", I'd say a merge is unlikely and I will continue maintaining my package. I wrote it to solve some specific problems I had and it IMO does that job quite well. It's nice if other people find it useful and I'm also proud to have ended up indirectly contributing to the official squashfs-tools development by providing new motivation :-). [1] https://sourceforge.net/p/squashfs/mailman/message/36298345/ https://sourceforge.net/p/squashfs/mailman/message/36298345/ [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965 [3] https://news.ycombinator.com/item?id=20518019 https://news.ycombinator.com/item?id=20518019 [4] https://lkml.org/lkml/2019/8/1/1154 https://lkml.org/lkml/2019/8/1/1154 [5] https://github.com/AgentD/squashfs-tools-ng/issues/10 https://github.com/AgentD/squashfs-tools-ng/issues/10 [6] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#29 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931965#29 [7] https://lkml.org/lkml/2019/8/1/565 https://lkml.org/lkml/2019/8/1/565