3 ms·

> Arguably, principle of least surprise is very Apple. Principle of least surprise is good engineering practice. The question is always whose surprise. Someone

> Arguably, principle of least surprise is very Apple. Principle of least surprise is good engineering practice. The question is always whose surprise. Someone who expects tar to behave like other UNIX systems is going to be surprised by this. Someone who expects tar on Apple to have perfect fidelity would be surprised by not-this. I increasingly feel like build systems should never be relying on any "native" utilities from the host system, and should instead be bringing them in via dependencies. You can't have this problem if your packaging system pulls in a specific portable `tar` library.

amarant 6mo ago
Nixos has a pretty solid solution to this issue: key your dependencies with checksums of the content. That way you get the best of both worlds: you always get the exact version you want, and you can share a copy of that exact version with other software that wants to use that exact version too!
- JoshTriplett 6mo ago
  Yeah, Nix-like distributions (e.g. guix, lix) do for Linux systems what some language package managers (e.g. cargo) do for individual projects.
- altairprime 6mo ago
  Are the xattr / chattr / umask checksums rolled into the main data fork content or are they hashed separately (or not at all)?
  - a_t48 6mo ago
    IIRC Nix is checksummed in the hash of the source of the content, not the results.
    - microtonal 6mo ago
      Hash of a normalization of the derivation, so this roughly means source, dependencies and the ‘build recipe’. The exception are fixed-output derivations, which are typically content-hashed. That said, a lot of work is done in content-addressed hashing, but AFAIK it’s not the default yet.
- dented42 6mo ago
  So it sounds like you don’t get the exact version you want because metadata is thrown away.
  - rrvsh 6mo ago
    It's a checksum not the content itself
  - amarant 6mo ago
    Curious, what is your software doing that it depends on specific metadata in your dependencies? What metadata do you require? Most files metadata is stuff like created timestamp, last edit timestamp, read/write/execute permissions.. I'm just trying to think of a case where metadata would be relevant in a dependency?
Joker_vD 6mo ago
> I increasingly feel like build systems should never be relying on any "native" utilities from the host system, and should instead be bringing them in via dependencies. Well, you see, while this, frankly, applies not just to build systems but to most of software, the consensus in the community of distro-maintainers is that it's actually wrong: you should use your system's package manager, and tools it can install, and let it fiddle with the ambient environment and give you that delicious "path dependency". And if your distro's packaging environment doesn't allow to do the things you need (e.g. being able to install both mongodb 3.8 and mongodb 5.0, ideally at the same time, but okay, I can keep running apt remove/install over and over, but I do need to check if my app correctly handled the wire protocol changes), well, that's your problem for desiring strange things.
crazygringo 6mo ago
> The question is always whose surprise. I think that the surprise of more data than expected is more desirable than the surprise of data loss. So in this case, it seems like the safe choice.
- dlenski 6mo ago
  Agreed. I usually hate on Apple, and its terribly ancient utilities and gratuitous incompatibility with modern Linux utilities, motivated by hatred of the GPL license. But in this case, I think what it's doing is… basically fine? "Tar should faithfully reproduce the semantics of the source filesystem" is a perfectly reasonable starting point. Ideally there would be a documented way to turn off the Apple-specific metadata with Apple's own tar, though.
  - saagarjha 6mo ago
    From tar(1): --no-mac-metadata (x mode only) Mac OS X specific. Do not archive or extract ACLs and extended file attributes using copyfile(3) in AppleDouble format. This is the reverse of --mac-metadata. and the default behavior if tar is run as non-root in x mode.
Arainach 6mo ago
Apple is always surprised that non-Apple devices exist. See: the permanent undismissable red icon to "finish setting up your Apple TV with your iPhone"
- simianparrot 6mo ago
  Apple can't control non-Apple devices. They can only control their own. So this makes perfect sense.
  - dwattttt 6mo ago
    They could control their own Apple TVs to allow that dialogue to be dismissed via the TV controls.
    - fingerlocks 6mo ago
      Agreed, but why not just finishing setting it up? Or do people own Apple TVs without iPhones? That never occurred to me since a large part of the value prop is phone integration
      - dwattttt 6mo ago
        Yes, I believe it's possible to buy an Apple TV without owning an iPhone.
      - jooize 6mo ago
        What integrations do you use? I can't really think of what I would miss on the Apple TV if I switched from iPhone. I rarely use AirPlay, disable Photos for in-house privacy reasons, and… oh yeah, the remote control for keyboard, volume, and navigation via iPhone is neat! I think the Apple TV is just a strong product on its own.
        fingerlocks 6mo ago
        I use screen mirroring, a lot. Guess I’m in the minority around here. Really nice projecting your phone on a massive OLED to multitask on the phone. Or even pair programming and conference calls you can mirror the phone to TV for the call while coding on the laptop. I use my Apple TV like it’s a big iPad stuck to the wall. Because that’s basically what it is. I honestly had no idea so many people just buy it to stream the same content on every other platform
Someone 6mo ago
> Someone who expects tar to behave like other UNIX systems is going to be surprised by this They shouldn’t. The GNU tar manual already shows this behavior. https://www.gnu.org/software/tar/manual/html_node/What-tar-Does.html#What-tar-Does https://www.gnu.org/software/tar/manual/html_node/What-tar-D...: Because the archive created by tar is capable of preserving file information and directory structure, tar is commonly used for performing full and incremental backups of disks” And yes, that same page also says: “You can create an archive on one system, transfer it to another system, and extract the contents there. This allows you to transport a group of files from one system to another.” > You can't have this problem if your packaging system pulls in a specific portable `tar` library. You can’t pull in specific portable stuff all the way down (not even when running in Docker or a VM), so that will decrease the risk, but it cannot completely remove it. As an example, I think GNU tar will happily include .DS_Store files in archives.
adrian_b 6mo ago
What should be really surprising for the users of UNIX-like operating systems is when they lose data because traditional UNIX utilities like cp, tar or cpio do not make complete copies of files, as one would expect from their description. What is worse is that these utilities do not give any warnings when they do not make complete copies. For cp, the root cause is that it has bad default options, while for tar and cpio the standard file formats cannot store the metadata of modern file systems. The various tar programs have their own different file format extensions to deal with modern file systems, which are guaranteed to work only when using the same tar program for both creation and extraction. The better tar programs implement both their own file format extensions and the file format extensions used by other popular tar programs. The author of the TFA has used some obsolete tar program, which is the cause for the surprising behavior that was seen. To avoid loss of data on Linux, I always use the PAX file format instead of tar or cpio, with the extensions implemented by "bsdtar --create --format=pax" from libarchive, and I always alias cp to '/bin/cp --no-dereference --recursive --one-file-system --preserve=all --strip-trailing-slashes --verbose --interactive', where cp has been built with extended attributes support.

amarant 6mo ago

Nixos has a pretty solid solution to this issue: key your dependencies with checksums of the content. That way you get the best of both worlds: you always get the exact version you want, and you can share a copy of that exact version with other software that wants to use that exact version too!

JoshTriplett 6mo ago
Yeah, Nix-like distributions (e.g. guix, lix) do for Linux systems what some language package managers (e.g. cargo) do for individual projects.
altairprime 6mo ago
Are the xattr / chattr / umask checksums rolled into the main data fork content or are they hashed separately (or not at all)?
- a_t48 6mo ago
  IIRC Nix is checksummed in the hash of the source of the content, not the results.
  - microtonal 6mo ago
    Hash of a normalization of the derivation, so this roughly means source, dependencies and the ‘build recipe’. The exception are fixed-output derivations, which are typically content-hashed. That said, a lot of work is done in content-addressed hashing, but AFAIK it’s not the default yet.
dented42 6mo ago
So it sounds like you don’t get the exact version you want because metadata is thrown away.
- rrvsh 6mo ago
  It's a checksum not the content itself
- amarant 6mo ago
  Curious, what is your software doing that it depends on specific metadata in your dependencies? What metadata do you require? Most files metadata is stuff like created timestamp, last edit timestamp, read/write/execute permissions.. I'm just trying to think of a case where metadata would be relevant in a dependency?

Joker_vD 6mo ago

> I increasingly feel like build systems should never be relying on any "native" utilities from the host system, and should instead be bringing them in via dependencies. Well, you see, while this, frankly, applies not just to build systems but to most of software, the consensus in the community of distro-maintainers is that it's actually wrong: you should use your system's package manager, and tools it can install, and let it fiddle with the ambient environment and give you that delicious "path dependency". And if your distro's packaging environment doesn't allow to do the things you need (e.g. being able to install both mongodb 3.8 and mongodb 5.0, ideally at the same time, but okay, I can keep running apt remove/install over and over, but I do need to check if my app correctly handled the wire protocol changes), well, that's your problem for desiring strange things.

crazygringo 6mo ago

> The question is always whose surprise. I think that the surprise of more data than expected is more desirable than the surprise of data loss. So in this case, it seems like the safe choice.

dlenski 6mo ago
Agreed. I usually hate on Apple, and its terribly ancient utilities and gratuitous incompatibility with modern Linux utilities, motivated by hatred of the GPL license. But in this case, I think what it's doing is… basically fine? "Tar should faithfully reproduce the semantics of the source filesystem" is a perfectly reasonable starting point. Ideally there would be a documented way to turn off the Apple-specific metadata with Apple's own tar, though.
- saagarjha 6mo ago
  From tar(1): --no-mac-metadata (x mode only) Mac OS X specific. Do not archive or extract ACLs and extended file attributes using copyfile(3) in AppleDouble format. This is the reverse of --mac-metadata. and the default behavior if tar is run as non-root in x mode.

Arainach 6mo ago

Apple is always surprised that non-Apple devices exist. See: the permanent undismissable red icon to "finish setting up your Apple TV with your iPhone"

simianparrot 6mo ago
Apple can't control non-Apple devices. They can only control their own. So this makes perfect sense.
- dwattttt 6mo ago
  They could control their own Apple TVs to allow that dialogue to be dismissed via the TV controls.
  - fingerlocks 6mo ago
    Agreed, but why not just finishing setting it up? Or do people own Apple TVs without iPhones? That never occurred to me since a large part of the value prop is phone integration
    - dwattttt 6mo ago
      Yes, I believe it's possible to buy an Apple TV without owning an iPhone.
    - jooize 6mo ago
      What integrations do you use? I can't really think of what I would miss on the Apple TV if I switched from iPhone. I rarely use AirPlay, disable Photos for in-house privacy reasons, and… oh yeah, the remote control for keyboard, volume, and navigation via iPhone is neat! I think the Apple TV is just a strong product on its own.
      - fingerlocks 6mo ago
        I use screen mirroring, a lot. Guess I’m in the minority around here. Really nice projecting your phone on a massive OLED to multitask on the phone. Or even pair programming and conference calls you can mirror the phone to TV for the call while coding on the laptop. I use my Apple TV like it’s a big iPad stuck to the wall. Because that’s basically what it is. I honestly had no idea so many people just buy it to stream the same content on every other platform

Someone 6mo ago

> Someone who expects tar to behave like other UNIX systems is going to be surprised by this They shouldn’t. The GNU tar manual already shows this behavior. https://www.gnu.org/software/tar/manual/html_node/What-tar-Does.html#What-tar-Does https://www.gnu.org/software/tar/manual/html_node/What-tar-D...: Because the archive created by tar is capable of preserving file information and directory structure, tar is commonly used for performing full and incremental backups of disks” And yes, that same page also says: “You can create an archive on one system, transfer it to another system, and extract the contents there. This allows you to transport a group of files from one system to another.” > You can't have this problem if your packaging system pulls in a specific portable `tar` library. You can’t pull in specific portable stuff all the way down (not even when running in Docker or a VM), so that will decrease the risk, but it cannot completely remove it. As an example, I think GNU tar will happily include .DS_Store files in archives.

adrian_b 6mo ago

What should be really surprising for the users of UNIX-like operating systems is when they lose data because traditional UNIX utilities like cp, tar or cpio do not make complete copies of files, as one would expect from their description. What is worse is that these utilities do not give any warnings when they do not make complete copies. For cp, the root cause is that it has bad default options, while for tar and cpio the standard file formats cannot store the metadata of modern file systems. The various tar programs have their own different file format extensions to deal with modern file systems, which are guaranteed to work only when using the same tar program for both creation and extraction. The better tar programs implement both their own file format extensions and the file format extensions used by other popular tar programs. The author of the TFA has used some obsolete tar program, which is the cause for the surprising behavior that was seen. To avoid loss of data on Linux, I always use the PAX file format instead of tar or cpio, with the extensions implemented by "bsdtar --create --format=pax" from libarchive, and I always alias cp to '/bin/cp --no-dereference --recursive --one-file-system --preserve=all --strip-trailing-slashes --verbose --interactive', where cp has been built with extended attributes support.