11 ms·

The trouble with symbolic links

227 points by jwilk 4y ago

smelbe 4y ago
There doesn't seem to be a way to batch together operations that involve walking through directories and symlinks to do something to a file. This seems to be a major source of complexity.
- klodolph 4y ago
  Could you elaborate? Seems like there’s a bunch of things that can’t be batched together on an ordinary file, without involving symlinks.
  - masklinn 4y ago
    What they (and TFA) are saying is that there is no transactional view of the FS. If you could work in “repeatable read” (only and always see the state of the FS before you started the transaction) symlink races wouldn’t be possible.
    - klodolph 4y ago
      Right, but there is no transactional view with or without symlinks.
      - masklinn 4y ago
        Without symlinks it doesn't matter, because by definition a symlink race requires a symlink to be involved.
- rwmj 4y ago
  I always thought Unix v7+ should have added some kind way to do atomic groups of syscalls, eg: begin_transaction (); lstat ("/path", ...); lstat ("/path/foo", ...); commit (); In Unix v7 mkdir was not a system call. It was a setuid program implemented using mknod + link. That was racy so the mkdir(2) system call was added. But it could have been solved more generally (and more elegantly) by adding transactions. It could also solve the whole thing with ending up with zero-length files because you didn't use the right incantation to update a file atomically on ext4 (https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a...).
  - bhawks 4y ago
    A general purpose transactional interface widene the error space to include cross process deadlocks / denial of service not to mention performance issues.
  - amaranth 4y ago
    Wasn't making userspace handle these kinds of things a big part of "worse is better"?
    - masklinn 4y ago
      Turns out when facing adversarial actors worse is just worse.
jmillikin 4y ago
IMO the article's conclusion is backwards. There's nothing wrong with symlinks when files are opened with openat(), since in principal the program (or controlling program) should always be in control of the filesystem layout. It's open() that causes problems, and complex interactions with symlinks in attacker-controlled directories are just one of them. The POSIX file API was designed before the concept of capability passing (arguably, before the concept of computer security in general). A modern replacement would look more like Fuchsia, where child processes are provided file access scoped to their parent process's authority. This same scoping can also be used within a process, for example to implement a server that can "self-chroot". > So, more functions following the pattern of openat() had to be created > [...] > Some are still missing, like getxattrat() and setxattrat(). The functions to get/set xattrs on a file descriptor (rather than a path) are fgetxattr() and fsetxattr(). They're not usable for the specific case of a file descriptor opened with O_PATH, but that restriction is both documented and reasonable -- O_PATH doesn't allow operations that inspect the state of the file itself, such as reading/writing. A better example might have been listxattr() vs flistxattr(), because the former works on a file without read permissions, but the latter fails on a descriptor opened with O_PATH. listxattr("xattr-chmod000.txt", NULL, 0) = 14 listxattr("xattr-chmod000.txt", "user.testattr\0", 14) = 14 getxattr("xattr-chmod000.txt", "user.testattr", NULL, 0) = -1 EACCES (Permission denied) vs openat(AT_FDCWD, "xattr-chmod000.txt", O_RDONLY|O_PATH) = 3 flistxattr(3, NULL, 0) = -1 EBADF (Bad file descriptor)
- jerf 4y ago
  "There's nothing wrong with symlinks when files are opened with openat(), since in principal the program (or controlling program) should always be in control of the filesystem layout. It's open() that causes problems, and complex interactions with symlinks in attacker-controlled directories are just one of them." One of my minor annoyances with new languages is the continued persistence of open-based file APIs, with openat APIs shoved off to the side if they are even implemented at all. If you start from scratch with an openat-based API, it's not even that hard; you basically get a file object, just one with some different attributes and methods (or appropriate local ideas), most of which you don't care about, and it's not that hard to work with if you start with that from day one. It can be quite hard to backport something deeply based on string-based path manipulation into the *at-based APIs, though. I haven't deeply studied it but you ought to be able to simulate an openat-based API on a conventional filesystem that doesn't support it. It may not immunize you to security issues, but at least the code ought to be as portable as any other code that starts getting detailed about its interactions with filesystems, which is already "kinda, not very, some elbow grease required"... it's not like the bar is sky high because all that stuff already works perfectly across all platforms and filesystems anyhow.
ziml77 4y ago
Interestingly, Windows actually did exactly what's proposed at the end when MS added them in Vista. To minimize the security issues with symlinks, you had to elevate to admin to create them. It was only during the life of Windows 10 that they even added the option to not have to elevate to create them. It was done specifically because symlinks are often shared across systems since they end up in places like git repos and npm packages: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ https://blogs.windows.com/windowsdeveloper/2016/12/02/symlin...
- faragon 4y ago
  You could create symlinks since Windows Vista without permission elevation, if having disabled the UAC ("User Account Control").
  - ziml77 4y ago
    Disabling UAC meant there was nowhere to elevate to because you just had admin rights all the time.
    - faragon 4y ago
      You're right, thank you.
- alerighi 4y ago
  There is an option (that is proposed when you install git) to allow normal users to create symlinks. The fact is that symlinks are very useful to a developer, and a lot of development tools make use of them.
- kasabali 4y ago
  > It was only during the life of Windows 10 that they even added the option to not have to elevate to create them Nah it was available from the start in local security policy settings. https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-vista/cc766301(v=ws.10)?redirectedfrom=MSDN#create-symbolic-links https://docs.microsoft.com/en-us/previous-versions/windows/i...
mongol 4y ago
"X is fundamentally broken" is a tired trope. To me, something is broken if it is no longer working as intended. It used to work, but now it does not - it is broken. If something works as intended, but its utility is limited, and it can be improved, it is not broken.
- joosters 4y ago
  Symlinks work as intended, but they cause a lot of unintended security vulnerabilities, i.e. they break lots of otherwise-functioning code. You can play with your words and redefine their meanings, but the vulnerabilities remain.
  - mongol 4y ago
    They have been around for 40+ years, they don't break code unless we are talking about code predating their introduction. It is not me playing with words, I am just pointing out a tired trope.
  - rob_c 4y ago
    > but they cause a lot of unintended security vulnerabilities No, bad coders on UNIX platforms do this. The code may be valid code, but if it's intending to support running on UNIX it should do it properly not assuming it's on a FAT32 filesystem in 2022. Or, even better, run the code you don't trust on fat32 filesystems, see how far that gets.
    - joosters 4y ago
      Don't victim blame. Do you really think that using open(), stat(), lstat() (!), realpath(), mkdir(), rename() etc etc etc is a sign of a bad coder? The problem is that the APIs set you up for unexpected failure, and even some of the provided workarounds to 'safely' handle symlinks don't do it well enough. In the case of symlinks, I think it's fair to blame the tools rather than the workman.
      - zx8080 4y ago
rob_c 4y ago
The trouble with symbolic links is users who don't understand/use linking coming from alternate platforms and developing on/for UNIX. This is a feature in the same way that shellsock was used as a feature for many years by experts. Thankfully I'm expecting something like POSIX to save us this time.
shadowgovt 4y ago
Don't hard links suffer from the issue that because they're actually links to a specific file, not path pointers, that you can replace the target file thinking you have updated something in the system and instead have stale hard links lying around, referencing the older version when you intended to replace the older version for all users? I think that in a hypothetical world where symlinks worked like hard links, we'd be swapping out the security complaints in this post for articles about how hard it is to upgrade a POSIX system properly, tools and tricks you can use to make sure you truly replaced all instances of a binary with a known vulnerability, and so on.
- jhallenworld 4y ago
  Hardlinks exist, so you already have this problem. Tar has to keep track of inode numbers to recreate hard links for example.
  - shadowgovt 4y ago
    What I mean is that it's pretty SOP to have a package that installs a new command to work by installing to /opt/my-package and then symlinking /usr/bin/my-cmd to /opt/my-package/my-cmd. In the absence of symlinks, that link would be hard and dpkg et. al would have to do package management by deleting the /usr/bin/my-cmd link and re-creating it instead of letting it ride, trusting that it will point to the correct thing when the update completes because the target bin will have changed.
jhallenworld 4y ago
Another problem with hard links: you can not hard link a directory. It would make ".." ambiguous. Also with hard link directories, you would want to be able "rmdir" non-empty directories, just to delete the link. But then you have the problem of reference loops, so how do you reclaim space reliably? You would need a garbage collection algorithm to find data not reachable by root.
- jmillikin 4y ago
  Whether directories can be hardlinked depends on the filesystem and OS. When macOS switched from HFS+ to APFS, one of the changes was that they dropped support for directory hardlinks. https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/archive/documentation/Fi...
  - jhallenworld 4y ago
    I'm not a macOS user, but these sure seem to add complexity: https://stackoverflow.com/questions/80875/what-is-the-unix-command-to-create-a-hardlink-to-a-directory-in-os-x https://stackoverflow.com/questions/80875/what-is-the-unix-c... The filesystem has to check for and disallow loops.
- nine_k 4y ago
  The GC algorithms for hardlinked directories can be the same as for harlinked files: reference counting.
  - jhallenworld 4y ago
    Only if there are no loops..
- samatman 4y ago
  A filesystem can afford a very, very slow, mark and sweep.
  - 4y ago
bhawks 4y ago
A posix filesystem by itself is not a defensible security perimeter. Symlinks introduce security problems but there are other sources as well. If you have a system where processes with different trust profiles share a common view of a file system you have to assume one can manipulate the filesystem state to subvert the other. Android has dealt with this via locking down and isolating apps to their own filesystems. Cross app communication and data sharing utilizes IPC primitives that have rich caller/Callie information that can be used to build capabilities and authn/authz checks. The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps.
- philsnow 4y ago
  > The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps. This, absolutely, but I think it's even worse than that; in my mind the value prop of k8s is twofold: declarative configuration and isolation that forces apps to be able to interact over a very small boundary, the network overlay.
js2 4y ago
I'm sorry symlinks are a thorn in Jeremy's side, but they are useful from a user's perspective. Hard links don't fill the same need. You can't normally hard link directories. If a file has multiple links, finding them all normally requires scanning the entire file system, so deleting a file now becomes harder. A file with multiple links doesn't have an obvious canonical path. As an example of all these issues, I manage a bunch of Mac build hosts with multiple Xcode versions installed. We only retain the most recent patch release of each major.minor version, but drop compatibility symlinks in place for the other versions. On macOS, an application is just a directory. So for example we'll have: Xcode-13.0.app -> Xcode-13.0.1.app Xcode-13.0.1.app From a simple "ls" it's obvious which versions are installed and which are just compatibility shims. Symlinks are just so damn convenient for use cases like this. Hard links don't cut the mustard here. So there are more reasons for symlinks than just "hard links are restricted to linking within the same filesystem", but yes, that too. Probably I'm just lacking imagination and there's a solution that offers the advantages of symlinks with none of the downsides, but in my experience, we see this sort of indirection all over the computing landscape, so it seems like there's a fundamental need for it.
- OnlyMortal 4y ago
  I take it an “alias” isn’t good in this case? An alias does follow a move of the original - usually.
  - yjftsjthsd-h 4y ago
    Interesting; how does that work under the hood?
    - OnlyMortal 4y ago
      https://en.m.wikipedia.org/wiki/Alias_(Mac_OS) https://en.m.wikipedia.org/wiki/Alias_(Mac_OS)
    - js2 4y ago
      It's macOS magic that requires HFS/APFS and doesn't work at the POSIX layer. It would not work for my use case, no. An alias is like a hybrid between a symbolic link and a hard link. Like a symbolic link, it's its own file type whose contents point to the original, but like a hard link it points to the original using its ID, not its path. So an alias works even if the original is moved, but it does not increase the original's link count and is its own distinct entity in the file system.
cosmiccatnap 4y ago
Features add complexity and robust features add robust complexity. Robust features that span a core component like filesystem handling span many utilities. Maybe we should do the same search on samba vulnerabilities have have him take a look in the mirror...
stjohnswarts 4y ago
Does anyone else read something like this Jeremy Allison gave a talk titled "The UNIX Filesystem API is profoundly broken: What to do about it?". and immediately shut down on the person saying it because he all know it's extremely hyperbolic given what is happening in reality? It's great to not like something and point out its flaws as far as you use them and "here's the great idea to fix those issues" but to try to offend you audience and the Unix community as the first words you see in a talk is a great way to have people shut off and think "oh great another neckbeard with an overinflated sense of self"
marcosdumay 4y ago
I really think that an "open the file as this user or group (also constrained to the current user's permissions)" option will solve the security problems better than the "open the file relative to this root" one. Or, failing that, an usable capability system (not SELinux). I don't think just checking for symlinks really solves anything. It may make your bugs harder to exploit, what is always good, but people use symlinks, so you have to support them, so the bugs will stay there.
forrestthewoods 4y ago
Hard linking files isn’t useful in my experience because it requires every tool working with that file to never delete it and recreate it. However that’s what exactly many tools do. So the only way to reliably “share” a file is with a symlink. At least this is true for my workflows.
alerighi 4y ago
Well there is the solution: work with file descriptors and not with paths. POSIX should be extended to make sure all functions that take a path has also the version that takes the file descriptor (to avoid the /proc/self/fd/%d hack, that is not portable to non-Linux OS that don't have /proc, and on Linux requires /proc to be mounted that is not always the case for example in sandboxes and chroots). You don't also only have problem with symlinks if you work with paths, but with any kind of paths. For example is wrong to check with the path if a file exists and then do something with it, because it can as well be deleted, modified, etc. You have to work with file descriptors, and use only one function (open) to resolve the path into a descriptor one time (that is also more efficient, since resolving a path is computationally expensive, especially on modern filesystems).
kazinator 4y ago
> An application running as root may try to check that /data/mydir is a regular directory (not a symlink) before opening the file /data/mydir/passwd. In between the time the program does the directory check and the file open, an attacker could replace the mydir directory with a symlink to /etc, and now the file opened is, unexpectedly, /etc/passwd. This is a kind of race condition known as a time-of-check-to-time-of-use (TOCTOU) race. That application is doing the wrong check; it should be validating that every component of the path is a directory which is only writable to root. First you stat("/"). OK, that is a directory and writable only to root: so no non-root process can put a symlink there. Next we check "/data". OK, that's a directory, and since we know / is owned by root and not world-writable, /data cannot be replaced by a symlink. And so on ... This can easily be made into a function like safe_path("/data/dir/path/to/mypasswd") which returns true only if no pathname component is something which a user other than the caller, or root, could tamper with to point to a different file. The open system call should have a flag for this, O_SAFE. That would alter the behavior of the name resolution function (traditionally, "namei") to do these checks along the path. The path could have symlinks, if they are not tamperable from the POV of the calling user. Typically superuser applications in Unix rely on filesystem structure. They set environment variables like PATH carefully, and stick to accessing data in known directories that had better be safe. If /data/mydir/passwd is something that is manipulated by a root application, then the system is misconfigured if any of these is writable to a non-root user: /, /data, /data/mydir or /data/mydir/passwd. If that is the case, you don't need symlinks to wreak havoc on the application. You can, for instance, write your own password into that password file and then falsely authenticate with that app.
yyyk2 4y ago
No, there is absolutely nothing broken about symbolic links. What is the issue here is accessing user files as root. That is inherently unsafe in POSIX and also affects hardlinks as well (even more so, since you have to "follow" hardlinks http://michael.orlitzky.com/articles/posix_hardlink_heartache.xhtml http://michael.orlitzky.com/articles/posix_hardlink_heartach...).
GoblinSlayer 4y ago
>Clients that have write access to the exported part of the file system under a share via SMB1 unix extensions or NFS can create symlinks that So, it's not symlinks broken, but SMB1 unix extensions broken when exposed to the world for symlink creation. AIU this features doesn't even serve windows interoperability. And if the author wanted to disable all symlinks altogether, what is the purpose of these extensions?
partialzero 4y ago
Maybe I'm being naive, but I don't get how "pathnames as a concept are now utterly broken in POSIX". Isn't this "merely" a problem that the resolution of the path name is dynamic and can change between inspection and use? Wouldn't a practice of resolving pathnames once (recursively, atomically, whatever) into an immutable, opaque, direct handle, such as file descriptor, before use solve this issue? I realize what I just said may be tantamount to "all file io ops taking path strings are broken" - but that seems like a problem with the initial API design, not with the concept of having a level of indirection in path name resolution itself.
- bityard 4y ago
  This is basically what I was going to say. The article spends a lot of time arguing that TOCTOU patterns introduce security vulnerabilities, which I think all programmers (should!) already know but then comes to the weird conclusion that we'd just be better off without symlinks instead of designing an API to work with them atomically. Kinda reminds me of how a lot of UX changes happen: "This really popular feature is a bit kludgy and hard to maintain, let's just rewrite the whole app without it! (Instead of doing the work required to make it not suck.)"
  - drdec 4y ago
    Almost all the TOCTOU examples given in the article could be modified not to involve symlinks and still be valid.
amluto 4y ago
I personally think that hard links should go away. I have trouble thinking of any use for hard links that isn’t better served by CoW links. Hard links have quite surprising properties with respect to chmod, they are awkward to handle in archival tools, and even reliably identifying them is awkward to impossible in general.
cryptonector 4y ago
The problem here is Linux.
bartimus 4y ago
I was copying a file today and thought: "Why is it duplicating the data? Why does it not just create a new second filename pointer in the filesystem?"
- WhatIsDukkha 4y ago
  Take a look at the cp man page - https://www.man7.org/linux/man-pages/man1/cp.1.html https://www.man7.org/linux/man-pages/man1/cp.1.html Look for "reflink" and you get your wish :)
  - bartimus 4y ago
    Very cool! :)
  - jwilk 4y ago
    FICLONE is the underlying ioctl: https://www.man7.org/linux/man-pages/man2/ioctl_ficlone.2.html https://www.man7.org/linux/man-pages/man2/ioctl_ficlone.2.ht... As of Linux 5.18, it's supported by btrfs, xfs, overlayfs and some networking filesystems.
kazinator 4y ago
Response in code: https://news.ycombinator.com/item?id=32201150 https://news.ycombinator.com/item?id=32201150

227 points by jwilk 4y ago

There doesn't seem to be a way to batch together operations that involve walking through directories and symlinks to do something to a file. This seems to be a major source of complexity.

klodolph 4y ago
Could you elaborate? Seems like there’s a bunch of things that can’t be batched together on an ordinary file, without involving symlinks.
- masklinn 4y ago
  What they (and TFA) are saying is that there is no transactional view of the FS. If you could work in “repeatable read” (only and always see the state of the FS before you started the transaction) symlink races wouldn’t be possible.
  - klodolph 4y ago
    Right, but there is no transactional view with or without symlinks.
    - masklinn 4y ago
      Without symlinks it doesn't matter, because by definition a symlink race requires a symlink to be involved.
rwmj 4y ago
I always thought Unix v7+ should have added some kind way to do atomic groups of syscalls, eg: begin_transaction (); lstat ("/path", ...); lstat ("/path/foo", ...); commit (); In Unix v7 mkdir was not a system call. It was a setuid program implemented using mknod + link. That was racy so the mkdir(2) system call was added. But it could have been solved more generally (and more elegantly) by adding transactions. It could also solve the whole thing with ending up with zero-length files because you didn't use the right incantation to update a file atomically on ext4 (https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/ https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a...).
- bhawks 4y ago
  A general purpose transactional interface widene the error space to include cross process deadlocks / denial of service not to mention performance issues.
- amaranth 4y ago
  Wasn't making userspace handle these kinds of things a big part of "worse is better"?
  - masklinn 4y ago
    Turns out when facing adversarial actors worse is just worse.

jmillikin 4y ago

IMO the article's conclusion is backwards. There's nothing wrong with symlinks when files are opened with openat(), since in principal the program (or controlling program) should always be in control of the filesystem layout. It's open() that causes problems, and complex interactions with symlinks in attacker-controlled directories are just one of them. The POSIX file API was designed before the concept of capability passing (arguably, before the concept of computer security in general). A modern replacement would look more like Fuchsia, where child processes are provided file access scoped to their parent process's authority. This same scoping can also be used within a process, for example to implement a server that can "self-chroot". > So, more functions following the pattern of openat() had to be created > [...] > Some are still missing, like getxattrat() and setxattrat(). The functions to get/set xattrs on a file descriptor (rather than a path) are fgetxattr() and fsetxattr(). They're not usable for the specific case of a file descriptor opened with O_PATH, but that restriction is both documented and reasonable -- O_PATH doesn't allow operations that inspect the state of the file itself, such as reading/writing. A better example might have been listxattr() vs flistxattr(), because the former works on a file without read permissions, but the latter fails on a descriptor opened with O_PATH. listxattr("xattr-chmod000.txt", NULL, 0) = 14 listxattr("xattr-chmod000.txt", "user.testattr\0", 14) = 14 getxattr("xattr-chmod000.txt", "user.testattr", NULL, 0) = -1 EACCES (Permission denied) vs openat(AT_FDCWD, "xattr-chmod000.txt", O_RDONLY|O_PATH) = 3 flistxattr(3, NULL, 0) = -1 EBADF (Bad file descriptor)

jerf 4y ago
"There's nothing wrong with symlinks when files are opened with openat(), since in principal the program (or controlling program) should always be in control of the filesystem layout. It's open() that causes problems, and complex interactions with symlinks in attacker-controlled directories are just one of them." One of my minor annoyances with new languages is the continued persistence of open-based file APIs, with openat APIs shoved off to the side if they are even implemented at all. If you start from scratch with an openat-based API, it's not even that hard; you basically get a file object, just one with some different attributes and methods (or appropriate local ideas), most of which you don't care about, and it's not that hard to work with if you start with that from day one. It can be quite hard to backport something deeply based on string-based path manipulation into the *at-based APIs, though. I haven't deeply studied it but you ought to be able to simulate an openat-based API on a conventional filesystem that doesn't support it. It may not immunize you to security issues, but at least the code ought to be as portable as any other code that starts getting detailed about its interactions with filesystems, which is already "kinda, not very, some elbow grease required"... it's not like the bar is sky high because all that stuff already works perfectly across all platforms and filesystems anyhow.

ziml77 4y ago

Interestingly, Windows actually did exactly what's proposed at the end when MS added them in Vista. To minimize the security issues with symlinks, you had to elevate to admin to create them. It was only during the life of Windows 10 that they even added the option to not have to elevate to create them. It was done specifically because symlinks are often shared across systems since they end up in places like git repos and npm packages: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ https://blogs.windows.com/windowsdeveloper/2016/12/02/symlin...

faragon 4y ago
You could create symlinks since Windows Vista without permission elevation, if having disabled the UAC ("User Account Control").
- ziml77 4y ago
  Disabling UAC meant there was nowhere to elevate to because you just had admin rights all the time.
  - faragon 4y ago
    You're right, thank you.
alerighi 4y ago
There is an option (that is proposed when you install git) to allow normal users to create symlinks. The fact is that symlinks are very useful to a developer, and a lot of development tools make use of them.
kasabali 4y ago
> It was only during the life of Windows 10 that they even added the option to not have to elevate to create them Nah it was available from the start in local security policy settings. https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-vista/cc766301(v=ws.10)?redirectedfrom=MSDN#create-symbolic-links https://docs.microsoft.com/en-us/previous-versions/windows/i...

mongol 4y ago

"X is fundamentally broken" is a tired trope. To me, something is broken if it is no longer working as intended. It used to work, but now it does not - it is broken. If something works as intended, but its utility is limited, and it can be improved, it is not broken.

joosters 4y ago
Symlinks work as intended, but they cause a lot of unintended security vulnerabilities, i.e. they break lots of otherwise-functioning code. You can play with your words and redefine their meanings, but the vulnerabilities remain.
- mongol 4y ago
  They have been around for 40+ years, they don't break code unless we are talking about code predating their introduction. It is not me playing with words, I am just pointing out a tired trope.
- rob_c 4y ago
  > but they cause a lot of unintended security vulnerabilities No, bad coders on UNIX platforms do this. The code may be valid code, but if it's intending to support running on UNIX it should do it properly not assuming it's on a FAT32 filesystem in 2022. Or, even better, run the code you don't trust on fat32 filesystems, see how far that gets.
  - joosters 4y ago
    Don't victim blame. Do you really think that using open(), stat(), lstat() (!), realpath(), mkdir(), rename() etc etc etc is a sign of a bad coder? The problem is that the APIs set you up for unexpected failure, and even some of the provided workarounds to 'safely' handle symlinks don't do it well enough. In the case of symlinks, I think it's fair to blame the tools rather than the workman.
    - zx8080 4y ago

rob_c 4y ago

The trouble with symbolic links is users who don't understand/use linking coming from alternate platforms and developing on/for UNIX. This is a feature in the same way that shellsock was used as a feature for many years by experts. Thankfully I'm expecting something like POSIX to save us this time.

shadowgovt 4y ago

Don't hard links suffer from the issue that because they're actually links to a specific file, not path pointers, that you can replace the target file thinking you have updated something in the system and instead have stale hard links lying around, referencing the older version when you intended to replace the older version for all users? I think that in a hypothetical world where symlinks worked like hard links, we'd be swapping out the security complaints in this post for articles about how hard it is to upgrade a POSIX system properly, tools and tricks you can use to make sure you truly replaced all instances of a binary with a known vulnerability, and so on.

jhallenworld 4y ago
Hardlinks exist, so you already have this problem. Tar has to keep track of inode numbers to recreate hard links for example.
- shadowgovt 4y ago
  What I mean is that it's pretty SOP to have a package that installs a new command to work by installing to /opt/my-package and then symlinking /usr/bin/my-cmd to /opt/my-package/my-cmd. In the absence of symlinks, that link would be hard and dpkg et. al would have to do package management by deleting the /usr/bin/my-cmd link and re-creating it instead of letting it ride, trusting that it will point to the correct thing when the update completes because the target bin will have changed.

jhallenworld 4y ago

Another problem with hard links: you can not hard link a directory. It would make ".." ambiguous. Also with hard link directories, you would want to be able "rmdir" non-empty directories, just to delete the link. But then you have the problem of reference loops, so how do you reclaim space reliably? You would need a garbage collection algorithm to find data not reachable by root.

jmillikin 4y ago
Whether directories can be hardlinked depends on the filesystem and OS. When macOS switched from HFS+ to APFS, one of the changes was that they dropped support for directory hardlinks. https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html https://developer.apple.com/library/archive/documentation/Fi...
- jhallenworld 4y ago
  I'm not a macOS user, but these sure seem to add complexity: https://stackoverflow.com/questions/80875/what-is-the-unix-command-to-create-a-hardlink-to-a-directory-in-os-x https://stackoverflow.com/questions/80875/what-is-the-unix-c... The filesystem has to check for and disallow loops.
nine_k 4y ago
The GC algorithms for hardlinked directories can be the same as for harlinked files: reference counting.
- jhallenworld 4y ago
  Only if there are no loops..
samatman 4y ago
A filesystem can afford a very, very slow, mark and sweep.
- 4y ago

bhawks 4y ago

A posix filesystem by itself is not a defensible security perimeter. Symlinks introduce security problems but there are other sources as well. If you have a system where processes with different trust profiles share a common view of a file system you have to assume one can manipulate the filesystem state to subvert the other. Android has dealt with this via locking down and isolating apps to their own filesystems. Cross app communication and data sharing utilizes IPC primitives that have rich caller/Callie information that can be used to build capabilities and authn/authz checks. The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps.

philsnow 4y ago
> The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps. This, absolutely, but I think it's even worse than that; in my mind the value prop of k8s is twofold: declarative configuration and isolation that forces apps to be able to interact over a very small boundary, the network overlay.

js2 4y ago

I'm sorry symlinks are a thorn in Jeremy's side, but they are useful from a user's perspective. Hard links don't fill the same need. You can't normally hard link directories. If a file has multiple links, finding them all normally requires scanning the entire file system, so deleting a file now becomes harder. A file with multiple links doesn't have an obvious canonical path. As an example of all these issues, I manage a bunch of Mac build hosts with multiple Xcode versions installed. We only retain the most recent patch release of each major.minor version, but drop compatibility symlinks in place for the other versions. On macOS, an application is just a directory. So for example we'll have: Xcode-13.0.app -> Xcode-13.0.1.app Xcode-13.0.1.app From a simple "ls" it's obvious which versions are installed and which are just compatibility shims. Symlinks are just so damn convenient for use cases like this. Hard links don't cut the mustard here. So there are more reasons for symlinks than just "hard links are restricted to linking within the same filesystem", but yes, that too. Probably I'm just lacking imagination and there's a solution that offers the advantages of symlinks with none of the downsides, but in my experience, we see this sort of indirection all over the computing landscape, so it seems like there's a fundamental need for it.

OnlyMortal 4y ago
I take it an “alias” isn’t good in this case? An alias does follow a move of the original - usually.
- yjftsjthsd-h 4y ago
  Interesting; how does that work under the hood?
  - OnlyMortal 4y ago
    https://en.m.wikipedia.org/wiki/Alias_(Mac_OS) https://en.m.wikipedia.org/wiki/Alias_(Mac_OS)
  - js2 4y ago
    It's macOS magic that requires HFS/APFS and doesn't work at the POSIX layer. It would not work for my use case, no. An alias is like a hybrid between a symbolic link and a hard link. Like a symbolic link, it's its own file type whose contents point to the original, but like a hard link it points to the original using its ID, not its path. So an alias works even if the original is moved, but it does not increase the original's link count and is its own distinct entity in the file system.

cosmiccatnap 4y ago

Features add complexity and robust features add robust complexity. Robust features that span a core component like filesystem handling span many utilities. Maybe we should do the same search on samba vulnerabilities have have him take a look in the mirror...

stjohnswarts 4y ago

Does anyone else read something like this Jeremy Allison gave a talk titled "The UNIX Filesystem API is profoundly broken: What to do about it?". and immediately shut down on the person saying it because he all know it's extremely hyperbolic given what is happening in reality? It's great to not like something and point out its flaws as far as you use them and "here's the great idea to fix those issues" but to try to offend you audience and the Unix community as the first words you see in a talk is a great way to have people shut off and think "oh great another neckbeard with an overinflated sense of self"

marcosdumay 4y ago

I really think that an "open the file as this user or group (also constrained to the current user's permissions)" option will solve the security problems better than the "open the file relative to this root" one. Or, failing that, an usable capability system (not SELinux). I don't think just checking for symlinks really solves anything. It may make your bugs harder to exploit, what is always good, but people use symlinks, so you have to support them, so the bugs will stay there.

forrestthewoods 4y ago

Hard linking files isn’t useful in my experience because it requires every tool working with that file to never delete it and recreate it. However that’s what exactly many tools do. So the only way to reliably “share” a file is with a symlink. At least this is true for my workflows.

alerighi 4y ago

Well there is the solution: work with file descriptors and not with paths. POSIX should be extended to make sure all functions that take a path has also the version that takes the file descriptor (to avoid the /proc/self/fd/%d hack, that is not portable to non-Linux OS that don't have /proc, and on Linux requires /proc to be mounted that is not always the case for example in sandboxes and chroots). You don't also only have problem with symlinks if you work with paths, but with any kind of paths. For example is wrong to check with the path if a file exists and then do something with it, because it can as well be deleted, modified, etc. You have to work with file descriptors, and use only one function (open) to resolve the path into a descriptor one time (that is also more efficient, since resolving a path is computationally expensive, especially on modern filesystems).

kazinator 4y ago

> An application running as root may try to check that /data/mydir is a regular directory (not a symlink) before opening the file /data/mydir/passwd. In between the time the program does the directory check and the file open, an attacker could replace the mydir directory with a symlink to /etc, and now the file opened is, unexpectedly, /etc/passwd. This is a kind of race condition known as a time-of-check-to-time-of-use (TOCTOU) race. That application is doing the wrong check; it should be validating that every component of the path is a directory which is only writable to root. First you stat("/"). OK, that is a directory and writable only to root: so no non-root process can put a symlink there. Next we check "/data". OK, that's a directory, and since we know / is owned by root and not world-writable, /data cannot be replaced by a symlink. And so on ... This can easily be made into a function like safe_path("/data/dir/path/to/mypasswd") which returns true only if no pathname component is something which a user other than the caller, or root, could tamper with to point to a different file. The open system call should have a flag for this, O_SAFE. That would alter the behavior of the name resolution function (traditionally, "namei") to do these checks along the path. The path could have symlinks, if they are not tamperable from the POV of the calling user. Typically superuser applications in Unix rely on filesystem structure. They set environment variables like PATH carefully, and stick to accessing data in known directories that had better be safe. If /data/mydir/passwd is something that is manipulated by a root application, then the system is misconfigured if any of these is writable to a non-root user: /, /data, /data/mydir or /data/mydir/passwd. If that is the case, you don't need symlinks to wreak havoc on the application. You can, for instance, write your own password into that password file and then falsely authenticate with that app.

yyyk2 4y ago

No, there is absolutely nothing broken about symbolic links. What is the issue here is accessing user files as root. That is inherently unsafe in POSIX and also affects hardlinks as well (even more so, since you have to "follow" hardlinks http://michael.orlitzky.com/articles/posix_hardlink_heartache.xhtml http://michael.orlitzky.com/articles/posix_hardlink_heartach...).

GoblinSlayer 4y ago

>Clients that have write access to the exported part of the file system under a share via SMB1 unix extensions or NFS can create symlinks that So, it's not symlinks broken, but SMB1 unix extensions broken when exposed to the world for symlink creation. AIU this features doesn't even serve windows interoperability. And if the author wanted to disable all symlinks altogether, what is the purpose of these extensions?

partialzero 4y ago

Maybe I'm being naive, but I don't get how "pathnames as a concept are now utterly broken in POSIX". Isn't this "merely" a problem that the resolution of the path name is dynamic and can change between inspection and use? Wouldn't a practice of resolving pathnames once (recursively, atomically, whatever) into an immutable, opaque, direct handle, such as file descriptor, before use solve this issue? I realize what I just said may be tantamount to "all file io ops taking path strings are broken" - but that seems like a problem with the initial API design, not with the concept of having a level of indirection in path name resolution itself.

bityard 4y ago
This is basically what I was going to say. The article spends a lot of time arguing that TOCTOU patterns introduce security vulnerabilities, which I think all programmers (should!) already know but then comes to the weird conclusion that we'd just be better off without symlinks instead of designing an API to work with them atomically. Kinda reminds me of how a lot of UX changes happen: "This really popular feature is a bit kludgy and hard to maintain, let's just rewrite the whole app without it! (Instead of doing the work required to make it not suck.)"
- drdec 4y ago
  Almost all the TOCTOU examples given in the article could be modified not to involve symlinks and still be valid.

amluto 4y ago

I personally think that hard links should go away. I have trouble thinking of any use for hard links that isn’t better served by CoW links. Hard links have quite surprising properties with respect to chmod, they are awkward to handle in archival tools, and even reliably identifying them is awkward to impossible in general.

cryptonector 4y ago

The problem here is Linux.

bartimus 4y ago

I was copying a file today and thought: "Why is it duplicating the data? Why does it not just create a new second filename pointer in the filesystem?"

WhatIsDukkha 4y ago
Take a look at the cp man page - https://www.man7.org/linux/man-pages/man1/cp.1.html https://www.man7.org/linux/man-pages/man1/cp.1.html Look for "reflink" and you get your wish :)
- bartimus 4y ago
  Very cool! :)
- jwilk 4y ago
  FICLONE is the underlying ioctl: https://www.man7.org/linux/man-pages/man2/ioctl_ficlone.2.html https://www.man7.org/linux/man-pages/man2/ioctl_ficlone.2.ht... As of Linux 5.18, it's supported by btrfs, xfs, overlayfs and some networking filesystems.

kazinator 4y ago

Response in code: https://news.ycombinator.com/item?id=32201150 https://news.ycombinator.com/item?id=32201150