6 ms·

Ex-Apple engineer here. This is, for better or worse, just the way Apple approaches this type of problem. From Apple's perspective, this is the way to preserve

Ex-Apple engineer here. This is, for better or worse, just the way Apple approaches this type of problem. From Apple's perspective, this is the way to preserve Finder / Gatekeeper / metadata semantics. It avoids silent data loss when round-tripping archives between Macs. This behavior also maintains consistency with copyfile(3) (as well as the Archive Utility behavior). Apple treats tar less like “portable Unix interchange” and more like “archive this filesystem object faithfully.” That is very Apple, and very libarchive. ;-) This is probably going to get worse (as Apple continues to add macOS-specific metadata), so your workaround is very helpful. I haven't tested it in a while, but at one point, setting the COPYFILE_DISABLE=1 env variable would disable the inclusion of macOS-specific metadata.

Terretta 5mo ago
Arguably, principle of least surprise is very Apple. If I point "tape archive" at a file system, I want that file system archived to tape. And so, tar does. If I don't, well, that's a fine option, and there's a fine option for that. So it's less of a "workaround" or something that "gets worse", than, "No, I don't really want a tape archive of this filesystem, only of some of it." And that's supported. That said, never seeing another .DS_Store should be a system-wide option!
- taftster 5mo ago
  > That said, never seeing another .DS_Store should be a system-wide option! Yes please.
  - ryandrake 5mo ago
    .DS_Store, .fseventsd, .Spotlight-V100, .Trashes, and ._this and ._that These can all die in a fire too, as far as I am concerned. macOS loves to treat the user's filesystem as its own personal garbage dump.
    - gerdesj 5mo ago
      thumbs.db and those weird MS alternative stream files for recording origination. filesystem attributes are for decorating files with meaning. Anything else that attempts to use filesystems in "interesting" ways is silly. Apple and MS really ought to consider why they do this sort of fragile, idiosyncratic nonsense.
      - Joker_vD 5mo ago
        But... thumbs.db is precisely not an "attempt to use filesystems in "interesting" ways" — it's literally a just hidden file with previews stored in it. Storing the preview in the alternative stream of the file with the picture itself would be "an interesting way".
        kstrauser 5mo ago
        Agreed. Where else would you put that stuff? It’s gotta go somewhere, and this is the least surprising place IMO. Anywhere else would have to be a parallel store that follows filesystem mounts and unmounts, renaming directories, etc so that it alway perfectly mirrors the thing it’s configuring.
        noisem4ker 5mo ago
        > Where else would you put that stuff? A "Centralized thumbnail cache" in the user profile folder, where it's been for a long while. https://en.wikipedia.org/wiki/Windows_thumbnail_cache https://en.wikipedia.org/wiki/Windows_thumbnail_cache > so that it alway perfectly mirrors Who cares? It's a cache.
        kstrauser 5mo ago
        That windows takes an approach does not mean it’s a good idea. And what about things like folder settings, such as whether to display is as a list or as icons, or how to sort it, etc? That’s more important than a thumbnail cache.
        5mo ago
- JoshTriplett 5mo ago
  > Arguably, principle of least surprise is very Apple. Principle of least surprise is good engineering practice. The question is always whose surprise. Someone who expects tar to behave like other UNIX systems is going to be surprised by this. Someone who expects tar on Apple to have perfect fidelity would be surprised by not-this. I increasingly feel like build systems should never be relying on any "native" utilities from the host system, and should instead be bringing them in via dependencies. You can't have this problem if your packaging system pulls in a specific portable `tar` library.
  - amarant 5mo ago
    Nixos has a pretty solid solution to this issue: key your dependencies with checksums of the content. That way you get the best of both worlds: you always get the exact version you want, and you can share a copy of that exact version with other software that wants to use that exact version too!
    - JoshTriplett 5mo ago
      Yeah, Nix-like distributions (e.g. guix, lix) do for Linux systems what some language package managers (e.g. cargo) do for individual projects.
    - altairprime 5mo ago
      Are the xattr / chattr / umask checksums rolled into the main data fork content or are they hashed separately (or not at all)?
      - a_t48 5mo ago
        IIRC Nix is checksummed in the hash of the source of the content, not the results.
        microtonal 5mo ago
        Hash of a normalization of the derivation, so this roughly means source, dependencies and the ‘build recipe’. The exception are fixed-output derivations, which are typically content-hashed. That said, a lot of work is done in content-addressed hashing, but AFAIK it’s not the default yet.
- saghm 5mo ago
  If you think that most people who run the tar command are assuming it will work like a tape archive, you'll probably be the one surprised
- adrian_b 5mo ago
  > I want that file system archived to tape. And so, tar does. The traditional UNIX tar and cpio utilities cannot archive the modern Linux file systems without loss of metadata. Most modern tar programs implement various file format extensions as a workaround for this, but the extensions may be incompatible between distinct tar programs and frequently they are very poorly documented. Some years in the past, libarchive was the only archiver available on Linux that guaranteed lossless backups for the Linux file systems, e.g. xfs or ext4 (and also lossless file transfers between Linux file systems and FreeBSD file systems). Therefore that is what I have been using on Linux since then. Presumably since then GNU tar and other tar programs should have caught up with it, but I have not verified this. Whichever tar program was used in TFA, it was an obsolete tar program, and that was the real problem, not that the archives had been created on an Apple computer.
matheusmoreira 5mo ago
It's a good attitude to have, in my opinion. Portability is overrated. Linux developers should be doing a lot more of this. We should be making everything work better for us without caring how it's going to impact other irrelevant platforms. Let the people who actually care about those platforms worry about such things.
- cozzyd 5mo ago
  It would at least be nice if there was a way to keep apple users from shitting all over the filesystem with remote mounts and ds_store files. Perhaps by automatically unmounting if one is detected.
  - bombcar 5mo ago
    At least if you're using ZFS as the backing store and Samba, you can set vfs objects = catia fruit streams_xattr and similar config options to use extended attributes.
  - seqastian 5mo ago
    defaults write com.apple.desktopservices DSDontWriteNetworkStores true
  - Maskawanian 5mo ago
    At least with Samba you can use the "veto files" and "delete veto files" global directives to deal with those, I personally use the following for veto files: /._*/.AppleDB/.AppleDouble/.AppleDesktop/:2eDS_Store/Network Trash Folder/Temporary Items/TheVolumeSettingsFolder/.@__thumb/.@__desc/:2e*/.@__qini/.Qsync/.@upload_cache/.qsync/.qsync_sn/.@qsys/.digest/ I understand that I may loose resource forks, but that isn't a problem for the use case of my server.
    - cozzyd 5mo ago
      unfortunately this is mostly people ssh mounting. I think I can probably write a ebpf rule to avoid writing them though. Or disconnect their sessions. Or modify the .DS_Store to change the finder background to something amusing.
jmclnx 5mo ago
To me, the big question is why Apple needs all these file attribute ? If the files are extracted OK, just ignore the errors :)
- bombcar 5mo ago
  Apple has had multiple streams per file since the very beginning, and it can store useful and necessary information (the latter is quite rare now, as most things have sane defaults, but losing the extended attributes can lose things that can be annoying).
hamasho 5mo ago
Funnily enough, I got the error message and asked Claude Code, and it replied; The warning can be suppressed by `--no-xattrs --no-mac-metadata`. then just edited the code as - tar czf dist.tar.gz dist + COPYFILE_DISABLE=1 tar czf dist.tar.gz dist
adrian_b 5mo ago
Yes, I completely disagree with TFA. The problem described in TFA is not specific to Apple, but the same problem appears when archiving any decent filesystem that has been designed during the last 3 decades and not a half of century ago, including all Linux file systems. The problem described in TFA is not caused by Apple, but by the author using an obsolete tar program and not being aware of this. The traditional tar file format cannot store a lot of the metadata that is contained in modern file systems (e.g. high resolution timestamps, access control lists, extended file attributes), so it is useless for such file systems. Most modern "tar" implementations have added extensions to the tar file format, to make it usable with modern file systems, such as Linux XFS or Linux EXT4. But many of these extensions are incompatible between themselves, so certain tar files can be fully extracted only with the same tar program that has created them. I strongly recommend against using the old tar or cpio file formats. Even with various extensions it is not guaranteed that they always work correctly. I always use only the pax file format, which has also required extensions in order to work with the modern file systems, but the pax extensions are cleaner than those for tar, because the file format is better designed. Libarchive, which was mentioned in TFA, is available in most Linux distributions or it can be built from source on any Linux computer. It provides an executable that is preferable to tar (better invoked as "bsdtar --format=pax") for the backup or transfer of any Linux files. I have not checked recently GNU tar or other tar programs available on Linux, and I hope that meanwhile they have been upgraded to be able to archive losslessly the Linux file systems, but some years ago that was not true, so using tar or cpio on Linux could easily corrupt the archived files.
- deleted 5mo ago
  [deleted]

Terretta 5mo ago

Arguably, principle of least surprise is very Apple. If I point "tape archive" at a file system, I want that file system archived to tape. And so, tar does. If I don't, well, that's a fine option, and there's a fine option for that. So it's less of a "workaround" or something that "gets worse", than, "No, I don't really want a tape archive of this filesystem, only of some of it." And that's supported. That said, never seeing another .DS_Store should be a system-wide option!

taftster 5mo ago
> That said, never seeing another .DS_Store should be a system-wide option! Yes please.
- ryandrake 5mo ago
  .DS_Store, .fseventsd, .Spotlight-V100, .Trashes, and ._this and ._that These can all die in a fire too, as far as I am concerned. macOS loves to treat the user's filesystem as its own personal garbage dump.
  - gerdesj 5mo ago
    thumbs.db and those weird MS alternative stream files for recording origination. filesystem attributes are for decorating files with meaning. Anything else that attempts to use filesystems in "interesting" ways is silly. Apple and MS really ought to consider why they do this sort of fragile, idiosyncratic nonsense.
    - Joker_vD 5mo ago
      But... thumbs.db is precisely not an "attempt to use filesystems in "interesting" ways" — it's literally a just hidden file with previews stored in it. Storing the preview in the alternative stream of the file with the picture itself would be "an interesting way".
      - kstrauser 5mo ago
        Agreed. Where else would you put that stuff? It’s gotta go somewhere, and this is the least surprising place IMO. Anywhere else would have to be a parallel store that follows filesystem mounts and unmounts, renaming directories, etc so that it alway perfectly mirrors the thing it’s configuring.
        noisem4ker 5mo ago
        > Where else would you put that stuff? A "Centralized thumbnail cache" in the user profile folder, where it's been for a long while. https://en.wikipedia.org/wiki/Windows_thumbnail_cache https://en.wikipedia.org/wiki/Windows_thumbnail_cache > so that it alway perfectly mirrors Who cares? It's a cache.
        kstrauser 5mo ago
        That windows takes an approach does not mean it’s a good idea. And what about things like folder settings, such as whether to display is as a list or as icons, or how to sort it, etc? That’s more important than a thumbnail cache.
      - 5mo ago
JoshTriplett 5mo ago
> Arguably, principle of least surprise is very Apple. Principle of least surprise is good engineering practice. The question is always whose surprise. Someone who expects tar to behave like other UNIX systems is going to be surprised by this. Someone who expects tar on Apple to have perfect fidelity would be surprised by not-this. I increasingly feel like build systems should never be relying on any "native" utilities from the host system, and should instead be bringing them in via dependencies. You can't have this problem if your packaging system pulls in a specific portable `tar` library.
- amarant 5mo ago
  Nixos has a pretty solid solution to this issue: key your dependencies with checksums of the content. That way you get the best of both worlds: you always get the exact version you want, and you can share a copy of that exact version with other software that wants to use that exact version too!
  - JoshTriplett 5mo ago
    Yeah, Nix-like distributions (e.g. guix, lix) do for Linux systems what some language package managers (e.g. cargo) do for individual projects.
  - altairprime 5mo ago
    Are the xattr / chattr / umask checksums rolled into the main data fork content or are they hashed separately (or not at all)?
    - a_t48 5mo ago
      IIRC Nix is checksummed in the hash of the source of the content, not the results.
      - microtonal 5mo ago
        Hash of a normalization of the derivation, so this roughly means source, dependencies and the ‘build recipe’. The exception are fixed-output derivations, which are typically content-hashed. That said, a lot of work is done in content-addressed hashing, but AFAIK it’s not the default yet.
saghm 5mo ago
If you think that most people who run the tar command are assuming it will work like a tape archive, you'll probably be the one surprised
adrian_b 5mo ago
> I want that file system archived to tape. And so, tar does. The traditional UNIX tar and cpio utilities cannot archive the modern Linux file systems without loss of metadata. Most modern tar programs implement various file format extensions as a workaround for this, but the extensions may be incompatible between distinct tar programs and frequently they are very poorly documented. Some years in the past, libarchive was the only archiver available on Linux that guaranteed lossless backups for the Linux file systems, e.g. xfs or ext4 (and also lossless file transfers between Linux file systems and FreeBSD file systems). Therefore that is what I have been using on Linux since then. Presumably since then GNU tar and other tar programs should have caught up with it, but I have not verified this. Whichever tar program was used in TFA, it was an obsolete tar program, and that was the real problem, not that the archives had been created on an Apple computer.

matheusmoreira 5mo ago

It's a good attitude to have, in my opinion. Portability is overrated. Linux developers should be doing a lot more of this. We should be making everything work better for us without caring how it's going to impact other irrelevant platforms. Let the people who actually care about those platforms worry about such things.

cozzyd 5mo ago
It would at least be nice if there was a way to keep apple users from shitting all over the filesystem with remote mounts and ds_store files. Perhaps by automatically unmounting if one is detected.
- bombcar 5mo ago
  At least if you're using ZFS as the backing store and Samba, you can set vfs objects = catia fruit streams_xattr and similar config options to use extended attributes.
- seqastian 5mo ago
  defaults write com.apple.desktopservices DSDontWriteNetworkStores true
- Maskawanian 5mo ago
  At least with Samba you can use the "veto files" and "delete veto files" global directives to deal with those, I personally use the following for veto files: /._*/.AppleDB/.AppleDouble/.AppleDesktop/:2eDS_Store/Network Trash Folder/Temporary Items/TheVolumeSettingsFolder/.@__thumb/.@__desc/:2e*/.@__qini/.Qsync/.@upload_cache/.qsync/.qsync_sn/.@qsys/.digest/ I understand that I may loose resource forks, but that isn't a problem for the use case of my server.
  - cozzyd 5mo ago
    unfortunately this is mostly people ssh mounting. I think I can probably write a ebpf rule to avoid writing them though. Or disconnect their sessions. Or modify the .DS_Store to change the finder background to something amusing.

jmclnx 5mo ago

To me, the big question is why Apple needs all these file attribute ? If the files are extracted OK, just ignore the errors :)

bombcar 5mo ago
Apple has had multiple streams per file since the very beginning, and it can store useful and necessary information (the latter is quite rare now, as most things have sane defaults, but losing the extended attributes can lose things that can be annoying).

hamasho 5mo ago

Funnily enough, I got the error message and asked Claude Code, and it replied; The warning can be suppressed by `--no-xattrs --no-mac-metadata`. then just edited the code as - tar czf dist.tar.gz dist + COPYFILE_DISABLE=1 tar czf dist.tar.gz dist

adrian_b 5mo ago

Yes, I completely disagree with TFA. The problem described in TFA is not specific to Apple, but the same problem appears when archiving any decent filesystem that has been designed during the last 3 decades and not a half of century ago, including all Linux file systems. The problem described in TFA is not caused by Apple, but by the author using an obsolete tar program and not being aware of this. The traditional tar file format cannot store a lot of the metadata that is contained in modern file systems (e.g. high resolution timestamps, access control lists, extended file attributes), so it is useless for such file systems. Most modern "tar" implementations have added extensions to the tar file format, to make it usable with modern file systems, such as Linux XFS or Linux EXT4. But many of these extensions are incompatible between themselves, so certain tar files can be fully extracted only with the same tar program that has created them. I strongly recommend against using the old tar or cpio file formats. Even with various extensions it is not guaranteed that they always work correctly. I always use only the pax file format, which has also required extensions in order to work with the modern file systems, but the pax extensions are cleaner than those for tar, because the file format is better designed. Libarchive, which was mentioned in TFA, is available in most Linux distributions or it can be built from source on any Linux computer. It provides an executable that is preferable to tar (better invoked as "bsdtar --format=pax") for the backup or transfer of any Linux files. I have not checked recently GNU tar or other tar programs available on Linux, and I hope that meanwhile they have been upgraded to be able to archive losslessly the Linux file systems, but some years ago that was not true, so using tar or cpio on Linux could easily corrupt the archived files.

deleted 5mo ago
[deleted]