6 ms·

It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just load

It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just loading the module and having the tree appear in /sys for you to point your compiler directly at. Also a bummer that it looks like you have to be running the kernel to get access to this— perhaps the archive could be marked off in the binary somehow so that there's a way to access it for non-running kernels (eg, the dkms use case of wanting to build the module for that kernel you just installed ahead of booting it for the first time). Overall binary size is a legit concern, but if this is going to be in an optionally-loaded module anyway, it seems weird to make that call just based on memory usage. I imagine most distro kernel builds will just disable this, since they already ship the headers in separate packages.

tych0 7y ago
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me. Hopefully someone picks it up :)
- hawski 7y ago
  Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for my use, but I think it would be generally useful. Sort list file format is funky. It's: "filename_string priority_integer", so it will not work for files with spaces. Also new lines, but that's far less common. It was not yet a problem for me and I can always patch it more and one can say, that people having files with spaces deserve it, though it's entirely different issue.
  - tych0 7y ago
    Looks like he might (?) be coming back: https://github.com/plougher/squashfs-tools/issues/54#issuecomment-507900346 https://github.com/plougher/squashfs-tools/issues/54#issueco... although the release he's talking about didn't happen. The last commit that wasn't an API refactoring in the kernel tree is a3f94cb99a854fa381fe7fadd97c4f61633717a5, which is from Aug 2, 2018.
  - usr1106 7y ago
    squashfs has horrible performance. All requests to the block layer are 512 Bytes. Other filesystems like ext4 make much bigger requests and perform much better in the end despite the compression of squashfs leading to lower overall data volume. Disclaimer: Measured 2 years ago on ARM32, emmc, with a 4.1(?) kernel.
    - cmurf 7y ago
      A big part of the problem might be xz decompression, that's been my discovery anyway. https://bugzilla.redhat.com/show_bug.cgi?id=1717728 https://bugzilla.redhat.com/show_bug.cgi?id=1717728
      - usr1106 7y ago
        I tried various compression algorithms. I don't think there was a CPU bottleneck, at least not with the less aggressive compressions. blktrace showed the difference compared to ext4, squashfs does all reads one by one block. It was in a previous job. I don't have access to the details anymore. And the kernel was not the newest. But squashfs looked unmaintained already then and that's what they're saying elsewhere in the discussion. So I fear nothing has changed.
    - plougher 7y ago
      Try using the SQUASHFS_4K_DEVBLK_SIZE config option next time By default Squashfs sets the dev block size (sb_min_blocksize) to 1K or the smallest block size supported by the block device (if larger). This, because blocks are packed together and unaligned in Squashfs, should reduce latency. This, however, gives poor performance on MTD NAND devices where the optimal I/O size is 4K (even though the devices can support smaller block sizes). Using a 4K device block size may also improve overall I/O performance for some file access patterns (e.g. sequential accesses of files in filesystem order) on all media. Setting this option will force Squashfs to use a 4K device block size by default. If unsure, say N.
- plougher 7y ago
  I'm the author and maintainer of Squashfs and I can assure you that Squashfs is not unmaintained. Over the last couple of years Squashfs has been stable, without requests for new features, and so work have been mostly only security improvements and bug fixes. But there is a big difference between that and claiming Squashfs is unmaintained. In fact I don't know why you're claiming it is unmaintained. Got an axe to grind perhaps?
  - tych0 7y ago
    I'm not claiming it's unmaintained. I'm saying that the maintainer is not active. You yourself mention elsewhere in this thread that you have little free time to maintain squashfs, so that seems fair. No axe to grind; it's a cool project and that's why we decided to use it. Thanks for your work on it, and we'll send patches if/when we have them :)

tych0 7y ago

squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me. Hopefully someone picks it up :)

hawski 7y ago
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for my use, but I think it would be generally useful. Sort list file format is funky. It's: "filename_string priority_integer", so it will not work for files with spaces. Also new lines, but that's far less common. It was not yet a problem for me and I can always patch it more and one can say, that people having files with spaces deserve it, though it's entirely different issue.
- tych0 7y ago
  Looks like he might (?) be coming back: https://github.com/plougher/squashfs-tools/issues/54#issuecomment-507900346 https://github.com/plougher/squashfs-tools/issues/54#issueco... although the release he's talking about didn't happen. The last commit that wasn't an API refactoring in the kernel tree is a3f94cb99a854fa381fe7fadd97c4f61633717a5, which is from Aug 2, 2018.
- usr1106 7y ago
  squashfs has horrible performance. All requests to the block layer are 512 Bytes. Other filesystems like ext4 make much bigger requests and perform much better in the end despite the compression of squashfs leading to lower overall data volume. Disclaimer: Measured 2 years ago on ARM32, emmc, with a 4.1(?) kernel.
  - cmurf 7y ago
    A big part of the problem might be xz decompression, that's been my discovery anyway. https://bugzilla.redhat.com/show_bug.cgi?id=1717728 https://bugzilla.redhat.com/show_bug.cgi?id=1717728
    - usr1106 7y ago
      I tried various compression algorithms. I don't think there was a CPU bottleneck, at least not with the less aggressive compressions. blktrace showed the difference compared to ext4, squashfs does all reads one by one block. It was in a previous job. I don't have access to the details anymore. And the kernel was not the newest. But squashfs looked unmaintained already then and that's what they're saying elsewhere in the discussion. So I fear nothing has changed.
  - plougher 7y ago
    Try using the SQUASHFS_4K_DEVBLK_SIZE config option next time By default Squashfs sets the dev block size (sb_min_blocksize) to 1K or the smallest block size supported by the block device (if larger). This, because blocks are packed together and unaligned in Squashfs, should reduce latency. This, however, gives poor performance on MTD NAND devices where the optimal I/O size is 4K (even though the devices can support smaller block sizes). Using a 4K device block size may also improve overall I/O performance for some file access patterns (e.g. sequential accesses of files in filesystem order) on all media. Setting this option will force Squashfs to use a 4K device block size by default. If unsure, say N.
plougher 7y ago
I'm the author and maintainer of Squashfs and I can assure you that Squashfs is not unmaintained. Over the last couple of years Squashfs has been stable, without requests for new features, and so work have been mostly only security improvements and bug fixes. But there is a big difference between that and claiming Squashfs is unmaintained. In fact I don't know why you're claiming it is unmaintained. Got an axe to grind perhaps?
- tych0 7y ago
  I'm not claiming it's unmaintained. I'm saying that the maintainer is not active. You yourself mention elsewhere in this thread that you have little free time to maintain squashfs, so that seems fair. No axe to grind; it's a cool project and that's why we decided to use it. Thanks for your work on it, and we'll send patches if/when we have them :)