5 ms·

Dissecting Apple's Sparse Image Format (ASIF)

163 points by supermatou 3mo ago

fragmede 3mo ago
I like a good jaunt with IDApro as much as the next RE, but my question is what does ASIF do that Qcow2 doesn't? My other question is why does it take so long to copy an app out of a dmg and into /Applications. Like, just change some pointers to pointers to data on disk and shit.
- donatj 3mo ago
  > what does ASIF do that Qcow2 doesn't Mount natively in macOS > why does it take so long to copy [...] out of a dmg Compression mostly. DMG contents can optionally be compressed using zlib, lzfse, or slow as molasses bzip2. Also Gatekeeper.
  - 1e1a 3mo ago
    Additionally, while I don't know much about APFS, I don't think it would be beneficial to point the extracted app to blocks that are also part of the dmg file, i.e. some copying has to happen anyway.
    - fragmede 3mo ago
      in a perfect theoretical filesystem, copy-on-write means copying is as cheap as moving a file, though uncompressing time makes sense.
      - galad87 3mo ago
        Of course, but that works only if the files are already in the same partition. A dmg is a virtual image, even if it's stored in the same partition, once mounted it acts like another partition.
      - xoa 3mo ago
        A perfect theoretical filesystem can still have subjective user configurable choices though right? Like case sensitivity, UTF normalization, checksum hash function, extra copies of data/metadata to store for redundancy/healing, etc (as well as compression/encryption). I think ZFS is a pretty strong real world example of a CoW FS, but you can still set a lot of different properties between sub-fs and then need to copy when you go between them to get the structural changes. Disk images are supposed to function as if they're attached storage I think, and have different properties from what FS you're running on boot or your home folder (which themselves can be different, I run my home folder on my main Mac off a NAS via iSCSI). I'm not sure any underlying FS would avoid a copy operation there in general?
ARTKILL 3mo ago
Worth noting ASIF's compression tradeoff also affects Spotlight indexing — since the content is opaque until mounted, you lose searchability on unmounted disk images that you'd get with a regular folder structure.
saagarjha 3mo ago
I have to admit that using C syntax as a string to parse something from Python is definitely a choice. I'm not even sure I would use C structs to lay things out in C…
- goeiedaggoeie 3mo ago
  in video/image space most code we deal with day to day is still C, lots more rust plugins in gstreamer ecosystem, but 90%+ still C
  - saagarjha 3mo ago
    Sure, but it's a greenfield Python script
  - flohofwoe 3mo ago
    The article is about disk images though :) But yeah, while I think the `cstruct` helper function to describe a binary data layout in Python is more elegant than the builtin alternatives, it would have been much less painful to just go with a minimal C command line program (or any other programming language where a struct directly maps to memory). Python and most other scripting languages have been built for manipulating text data, but suck when working with binary data.
- quietbritishjim 3mo ago
  I asked an LLM to rewrite it for me using the Python built-in struct module, and it gave me this: import sys import struct from collections import namedtuple # Bake the layout once into a reusable, precompiled object. HEADER = struct.Struct(">4sIIIQQ16sQQIIII") # struct only knows positions, not names — pair it with a namedtuple # to recover the named-field access that cstruct gives you for free. Header = namedtuple("Header", [ "magic", "field4", "field8", "fieldC", "field10", "field18", "field20", "field30", "field38", "field40", "field44", "field48", "field4C", ]) with open(sys.argv[1], "rb") as fh: header = Header._make(HEADER.unpack(fh.read(HEADER.size))) print(header) To me, this seems significantly less readable... less Pythonic, even. The printed output is also less readable.
dcrazy 3mo ago
More info on how ASIF differs from the decades-old sparseimage format: https://news.ycombinator.com/item?id=44259132 https://news.ycombinator.com/item?id=44259132
albertzeyer 3mo ago
Small self-advertisement: as an alternative to dissect.cstruct, a fun side-project of mine (C parser + C interpreter in Python) can do a very similar thing: https://github.com/albertz/PyCParser/blob/master/demos/dissect_cstruct.py https://github.com/albertz/PyCParser/blob/master/demos/disse...
ComputerGuru 3mo ago
Image as in “filesystem snapshot” not as in “media file”.
- ValentineC 3mo ago
  Speaking of images (media file), I found out that iCloud double-counts storage space for rotated photos, so I need to subscribe to a higher tier just because the iPhone often can't get orientation right, especially for food photos. Would be nice if they moved to storing diff sidecars.

163 points by supermatou 3mo ago

I like a good jaunt with IDApro as much as the next RE, but my question is what does ASIF do that Qcow2 doesn't? My other question is why does it take so long to copy an app out of a dmg and into /Applications. Like, just change some pointers to pointers to data on disk and shit.

donatj 3mo ago
> what does ASIF do that Qcow2 doesn't Mount natively in macOS > why does it take so long to copy [...] out of a dmg Compression mostly. DMG contents can optionally be compressed using zlib, lzfse, or slow as molasses bzip2. Also Gatekeeper.
- 1e1a 3mo ago
  Additionally, while I don't know much about APFS, I don't think it would be beneficial to point the extracted app to blocks that are also part of the dmg file, i.e. some copying has to happen anyway.
  - fragmede 3mo ago
    in a perfect theoretical filesystem, copy-on-write means copying is as cheap as moving a file, though uncompressing time makes sense.
    - galad87 3mo ago
      Of course, but that works only if the files are already in the same partition. A dmg is a virtual image, even if it's stored in the same partition, once mounted it acts like another partition.
    - xoa 3mo ago
      A perfect theoretical filesystem can still have subjective user configurable choices though right? Like case sensitivity, UTF normalization, checksum hash function, extra copies of data/metadata to store for redundancy/healing, etc (as well as compression/encryption). I think ZFS is a pretty strong real world example of a CoW FS, but you can still set a lot of different properties between sub-fs and then need to copy when you go between them to get the structural changes. Disk images are supposed to function as if they're attached storage I think, and have different properties from what FS you're running on boot or your home folder (which themselves can be different, I run my home folder on my main Mac off a NAS via iSCSI). I'm not sure any underlying FS would avoid a copy operation there in general?

ARTKILL 3mo ago

Worth noting ASIF's compression tradeoff also affects Spotlight indexing — since the content is opaque until mounted, you lose searchability on unmounted disk images that you'd get with a regular folder structure.

saagarjha 3mo ago

I have to admit that using C syntax as a string to parse something from Python is definitely a choice. I'm not even sure I would use C structs to lay things out in C…

goeiedaggoeie 3mo ago
in video/image space most code we deal with day to day is still C, lots more rust plugins in gstreamer ecosystem, but 90%+ still C
- saagarjha 3mo ago
  Sure, but it's a greenfield Python script
- flohofwoe 3mo ago
  The article is about disk images though :) But yeah, while I think the `cstruct` helper function to describe a binary data layout in Python is more elegant than the builtin alternatives, it would have been much less painful to just go with a minimal C command line program (or any other programming language where a struct directly maps to memory). Python and most other scripting languages have been built for manipulating text data, but suck when working with binary data.
quietbritishjim 3mo ago
I asked an LLM to rewrite it for me using the Python built-in struct module, and it gave me this: import sys import struct from collections import namedtuple # Bake the layout once into a reusable, precompiled object. HEADER = struct.Struct(">4sIIIQQ16sQQIIII") # struct only knows positions, not names — pair it with a namedtuple # to recover the named-field access that cstruct gives you for free. Header = namedtuple("Header", [ "magic", "field4", "field8", "fieldC", "field10", "field18", "field20", "field30", "field38", "field40", "field44", "field48", "field4C", ]) with open(sys.argv[1], "rb") as fh: header = Header._make(HEADER.unpack(fh.read(HEADER.size))) print(header) To me, this seems significantly less readable... less Pythonic, even. The printed output is also less readable.

dcrazy 3mo ago

More info on how ASIF differs from the decades-old sparseimage format: https://news.ycombinator.com/item?id=44259132 https://news.ycombinator.com/item?id=44259132

albertzeyer 3mo ago

Small self-advertisement: as an alternative to dissect.cstruct, a fun side-project of mine (C parser + C interpreter in Python) can do a very similar thing: https://github.com/albertz/PyCParser/blob/master/demos/dissect_cstruct.py https://github.com/albertz/PyCParser/blob/master/demos/disse...

ComputerGuru 3mo ago

Image as in “filesystem snapshot” not as in “media file”.

ValentineC 3mo ago
Speaking of images (media file), I found out that iCloud double-counts storage space for rotated photos, so I need to subscribe to a higher tier just because the iPhone often can't get orientation right, especially for food photos. Would be nice if they moved to storing diff sidecars.