5 ms·

Squashfs binary format

51 points by st_goliath 4y ago

gtirloni 4y ago
Also related: https://news.ycombinator.com/item?id=32216275 https://news.ycombinator.com/item?id=32216275
motiejus 4y ago
I wrote undocker[1], which converts a docker "container" to a rootfs tarball. I really wish docker container format were documented as well as squashfs here. Surprising, given Docker Inc should have had enough resources to make their most important format well documented. :) Anyway, undocker requires extracting the archive before using it. Squashfs wouldn't. Perhaps I will write a docker2squash, so we can mount and run these images immediately after download. [1]: https://git.sr.ht/~motiejus/undocker https://git.sr.ht/~motiejus/undocker
- st_goliath 4y ago
  > Perhaps I will write a docker2squash, so we can mount and run these images immediately after download. Well, if you already have a rootfs tarball, I guess the bulk of the work is done. The tarball can be directly piped into tar2sqfs[1] or sqfstar[2]. [1] https://manpages.debian.org/unstable/squashfs-tools-ng/tar2sqfs.1.en.html https://manpages.debian.org/unstable/squashfs-tools-ng/tar2s... [2] https://manpages.debian.org/testing/squashfs-tools/sqfstar.1.en.html https://manpages.debian.org/testing/squashfs-tools/sqfstar.1...
  - motiejus 4y ago
    Nice, thanks! I won't. :)
- chriswarbo 4y ago
  > I wrote undocker[1], which converts a docker "container" to a rootfs tarball I thought a "container" (or, more precisely, a "container image"; or even more precisely, a "container image layer") is a tarball? I've been creating multi-layer container images using `tar` (plus `sha256sum` and `jq` for the corresponding JSON "manifests"), and it works fine e.g. on AWS ECS.
  - guipsp 4y ago
    Can you detail further? - Personally, this seems like insanity so I'm curious what lead you down this path instead of using the more common tools like docker compose or buildah.
    - chriswarbo 4y ago
      > instead of using the more common tools like docker compose or buildah How are those tools "more common" than GNU tar? > I'm curious what lead you down this path I had an executable; I wanted to run it on ECS; so I ran the `tar` command. Pretty simple really :)
  - motiejus 4y ago
    A tarball of tarballs, layered using barely documented rules (e.g. you can add a directory, remove it on upper layer, and re-add something else on top). Constructing them is easier than unpacking. :)
    - chriswarbo 4y ago
      Ah, so you're re-creating the results of overlay-mounting the layers? I'm definitely ignorant of the exact rules for those; I only use them to split up large images into a "mostly static" layer (e.g. dependencies, which may be large) and a "subject to change" layer (i.e. our code, which will be different for each version)
chme 4y ago
So they are using u32 for their mtime, which is better than i32 because of the y2038 problem... However that just means they have just another odd 70 years left after 2038, right?
- gnabgib 4y ago
  Sure, but it's just for the archive-mod time in second resolution, by 2106 you'll probably have moved to a different filesystem, a newer version of SquashFS, or the ground.
secure 4y ago
Plugging this here in case it’s useful to anyone else working with squashfs: I wrote a squashfs reader/writer in Go: https://github.com/distr1/distri/tree/master/internal/squashfs https://github.com/distr1/distri/tree/master/internal/squash... (part of the https://distr1.org/ https://distr1.org/ project)

51 points by st_goliath 4y ago

Also related: https://news.ycombinator.com/item?id=32216275 https://news.ycombinator.com/item?id=32216275

I wrote undocker[1], which converts a docker "container" to a rootfs tarball. I really wish docker container format were documented as well as squashfs here. Surprising, given Docker Inc should have had enough resources to make their most important format well documented. :) Anyway, undocker requires extracting the archive before using it. Squashfs wouldn't. Perhaps I will write a docker2squash, so we can mount and run these images immediately after download. [1]: https://git.sr.ht/~motiejus/undocker https://git.sr.ht/~motiejus/undocker

st_goliath 4y ago
> Perhaps I will write a docker2squash, so we can mount and run these images immediately after download. Well, if you already have a rootfs tarball, I guess the bulk of the work is done. The tarball can be directly piped into tar2sqfs[1] or sqfstar[2]. [1] https://manpages.debian.org/unstable/squashfs-tools-ng/tar2sqfs.1.en.html https://manpages.debian.org/unstable/squashfs-tools-ng/tar2s... [2] https://manpages.debian.org/testing/squashfs-tools/sqfstar.1.en.html https://manpages.debian.org/testing/squashfs-tools/sqfstar.1...
- motiejus 4y ago
  Nice, thanks! I won't. :)
chriswarbo 4y ago
> I wrote undocker[1], which converts a docker "container" to a rootfs tarball I thought a "container" (or, more precisely, a "container image"; or even more precisely, a "container image layer") is a tarball? I've been creating multi-layer container images using `tar` (plus `sha256sum` and `jq` for the corresponding JSON "manifests"), and it works fine e.g. on AWS ECS.
- guipsp 4y ago
  Can you detail further? - Personally, this seems like insanity so I'm curious what lead you down this path instead of using the more common tools like docker compose or buildah.
  - chriswarbo 4y ago
    > instead of using the more common tools like docker compose or buildah How are those tools "more common" than GNU tar? > I'm curious what lead you down this path I had an executable; I wanted to run it on ECS; so I ran the `tar` command. Pretty simple really :)
- motiejus 4y ago
  A tarball of tarballs, layered using barely documented rules (e.g. you can add a directory, remove it on upper layer, and re-add something else on top). Constructing them is easier than unpacking. :)
  - chriswarbo 4y ago
    Ah, so you're re-creating the results of overlay-mounting the layers? I'm definitely ignorant of the exact rules for those; I only use them to split up large images into a "mostly static" layer (e.g. dependencies, which may be large) and a "subject to change" layer (i.e. our code, which will be different for each version)

chme 4y ago

So they are using u32 for their mtime, which is better than i32 because of the y2038 problem... However that just means they have just another odd 70 years left after 2038, right?

gnabgib 4y ago
Sure, but it's just for the archive-mod time in second resolution, by 2106 you'll probably have moved to a different filesystem, a newer version of SquashFS, or the ground.

secure 4y ago

Plugging this here in case it’s useful to anyone else working with squashfs: I wrote a squashfs reader/writer in Go: https://github.com/distr1/distri/tree/master/internal/squashfs https://github.com/distr1/distri/tree/master/internal/squash... (part of the https://distr1.org/ https://distr1.org/ project)