8 ms·

PATH_MAX Is Tricky

67 points by eklitzke 10y ago

dom0 10y ago
Contrary to some previous entries [1][2] by The Author this one is accurate. [1] https://news.ycombinator.com/item?id=13896675 https://news.ycombinator.com/item?id=13896675 [2] https://news.ycombinator.com/item?id=13935256 https://news.ycombinator.com/item?id=13935256
derefr 10y ago
> The problem is that you can’t meaningfully define a constant like this in a header file. The maximum path size is actually to be something like a filesystem limitation, or at the very least a kernel parameter. AFAIK, "paths" aren't a thing filesystems think about. As far as a filesystem driver is concerned, paths are either—for open/read/write/etc.—plain inodes (a uint64), or—for directory-manipulation calls—an inode plus a ptrdiff_t to index into the dirent list†. The only things that care about NAME_MAX are lookup(2) [get inode given {inode, dirent}], and link(2) [put inode in {inode, dirent}]. So it's really only the kernel, through its syscall interface, that cares about paths—and so PATH_MAX is just a representation of the maximum size of a path the kernel is willing to accept in those syscalls. As if they each had a statically-allocated path[PATH_MAX] buffer your path got copied into. † Writing a FUSE filesystem is a great way to learn about what the kernel thinks a filesystem is. It's very different from the userland perspective. For example, from a filesystem driver's perspective, "file descriptors" don't exist! Instead, read(2) and write(2) calls are stateless, each call getting passed a kernel-side io-handle struct that must get re-evaluated for matching permissions on each IO operation. (You can do some up-front evaluation during the open(2) call, but, given that a file's permissions might change while you have a descriptor open to it, there's not much point.)
- codys 10y ago
  One of the things filesystems provide is a mapping from paths to data (files, directories, etc). How they provide this varies (tree of path elements, etc), but saying they don't think about paths (and implying they only think about path elements) is misleading. Nothing prevents a file system from simply putting all it's path to data mappings into a hash table (except the slight difficulty in implementing some common filesystem operations).
  - derefr 10y ago
    Maybe on different OSes, but like I said above, on Linux, many operations will literally just pass a filesystem driver an inode. Your hypothetical driver would need to keep a double-mapping: from inodes to paths, and then from paths to whatever else. At which point, why bother with the second mapping? It's just slowing things down.
    - codys 10y ago
      The typical (generalized) setup is paths -> inodes (called "refers to data" or ""something"" in my other comments to avoid using the weird terminology we've somehow adapted for naming file system things) and inodes -> actual data on the disk. Whether you actually have that setup on disk is up to the fs format, but regardless of the format it is typically useful (and as you note, generic file system interfaces like the one in linux do require this) to have this extra indirection when tracking things on the software side (ie: not necessarily the storage side, though many file systems do have this type of indirection because it is also useful on the storage side). > Your hypothetical driver would need to keep a double-mapping: from inodes to paths, and then from paths to whatever else. As I've noted in another comment, inodes (`struct inode` in linux) typically contain a lot of information. No additional mapping would be required.
      - derefr 10y ago
        My point was that paths are part of Linux VFS, and that lookup is already done and invisible when the calls reach the FS driver. So there's no point in implementing an FS driver in terms of paths. It's redundant working-backward from something that was already done for you.
    - dom0 10y ago
      > Maybe on different OSes, Perhaps on a mainframe OS or something that doesn't have a VFS, but all VFS pretty much work the same way conceptually (BSD/nix, Linux, Windows).
  - loeg 10y ago
    That would make for some very expensive lookup operations on subdirectories (you have to scan the entire hash table) and would entirely preclude hardlinks. So yeah, usually people do not write byzantine filesystems :-).
- jstimpfle 10y ago
  > For example, from a filesystem driver's perspective, "file descriptors" don't exist! Instead, read(2) and write(2) calls are stateless That seems hard to believe. It would inefficient to look for a file for each e.g. write(). I guess a filesystem defines its own implementation-defined open-file handle and the kernel translates that into an (integer) FD. > each call getting passed a kernel-side io-handle struct that must get re-evaluated for matching permissions on each IO operation. (You can do some up-front evaluation during the open(2) call, but, given that a file's permissions might change while you have a descriptor open to it, there's not much point.) That's not how it works. File modes are checked when you open the file. Once you have an open file (a file handle), the file modes no longer matter. File modes are more a "PATH" thing (there aren't per-filesystem variations and I assume permission checking is not done in file system code), although the file system must allocate the space to save the file mode bits and implement the VFS API.
  - codys 10y ago
    Yep, this is misleading. "file descriptors" taken _literally_ don't exist (they are just the interface exposed to user space programs). Linux uses `struct file` internally as the generic file tracking, and multiple file descriptors can refer to a single `struct file`. `struct file` in turn refers to a `struct inode`, which represents an actual "something" on disk. Each file system driver stores a bunch of info in the `struct inode`. In addition to that tracking, there is generally a bunch of fs-specific data structures to handle lookup of names (in the typical case, these are essentially cached versions of what is stored on disk).
  - loeg 10y ago
    > That seems hard to believe. It would inefficient to look for a file for each e.g. write(). I guess a filesystem defines its own implementation-defined open-file handle and the kernel translates that into an (integer) FD. At least in BSD, the syscall layer calls into the kernel virtual filesystem, which translates file descriptors into file objects. File objects have fo_read/fo_write methods implemented by different kinds of file object. Files backed by filesystems have an associated "vnode" (virtual inode object). Then the fo_read/fo_write layer invokes the VOP_READ or VOP_WRITE (for example) method on the vnode. VOP ("vnode operation") methods are implemented by individual filesystems. Multiple user-level file descriptors can refer to a single vnode. The filesystem's inode object hangs off a pointer from the generic vnode. Vnodes are unique (1:1) per real file (inode). So: read(fd, ...) -> sys_read -> kern_readv -> fget() (translate fd number into struct file object; check that the descriptor was opened read/write as appropriate for the operation) dofileread() -> fo_read() -> vn_io_fault() (fo_read method for vnodes) -> vn_read -> VOP_READ(vnode, uio, ioflags, ucred) (vnode for the file, an io descriptor ("uio"), io flags, and user credentials) So yeah, at the filesystem layet (VOP_READ), you no longer have the fd (or even the file object) in the BSD model. Linux's model is similar, but not completely identical. They call "vnodes" "inodes" instead, and may not have the intermediary file layer.
josteink 10y ago
That's tricky indeed. And doing things properly seems quite involved. For now I'll keep my limits.h. At least until I get a real-world bug-reports telling me this is causing real-world issues :)
loeg 10y ago
The GNU Hurd approach to PATH_MAX is to set it to something ridiculous like SIZE_MAX, something that cannot possible be allocated, to illustrate to programmers that it is a fiction. I don't think that's necessarily the best approach, but it matches reality more closely than typical Linux/BSD values (1024 or 4096).
- the_mitsuhiko 10y ago
  That seems like a bad idea because people will then use inconsistent max path lengths.
  - jerf 10y ago
    What is the correct consistent max path length?
    - jstimpfle 10y ago
      It simplifies implementations. For example, typically you need to be able to store paths in a consecutive area of memory or storage. Knowing that paths don't exceed a certain size means you can know that a particular allocation strategy (like static allocation, for a simple example) will be sufficient.
      - klodolph 10y ago
        Mostly it's for stuff like readlink(), which returns a path in a user-supplied buffer. How big should the buffer be? The Linux man page actually suggests doing lstat() to size the buffer, with a note that this is susceptible to a race condition!
        dom0 10y ago
        readlink(2) takes a buffer and the buffer's size, so it's safe. You just enter a race-controlled loop, as always.
jstimpfle 10y ago
The essence of the article: PATH_MAX applies to the syscall interface. It's not related to file systems. Paths aren't a file system thing, but simply a convenient means of addressing files. Basically they are URLs for local resources. And that totally makes sense once you understand that files are basically "objects" (in the OO sense) identified by inodes instead of memory addresses. A file system implements the graph of these objects (linked by special file objects called directories). The fact that one can cross file system boundaries using file paths also indicates that file paths are none of a file system's business.
- masklinn 10y ago
  > It's not related to file systems. Paths aren't a file system thing, but simply a convenient means of addressing files. > The fact that one can cross file system boundaries using file paths also indicates that file paths are none of a file system's business. The filesystem knows about file names, stores them, and puts limits on them (often 255 code units though some are lower — FAT16's 8.3, HFS's 31 — and some are higher — Reiser4's 3976 bytes). A file path is nothing but a concatenation of a bunch of file names and separators ergo file paths are, in fact, an FS's business. And while that's mostly fallen out of style there are still length-limited-path filesystems: ISO-9660 and UDF for instance.
  - jstimpfle 10y ago
    > A file path is nothing but a concatenation of a bunch of file names and separators ergo file paths are, in fact, an FS's business. This is a non sequitur.
problems 10y ago
Windows has a similarly disastrous situation where most tools and APIs follow MAX_PATH - which is defined to be 260 chars. But that doesn't affect the actual filesystem or syscall interface, just common APIs and tools. This makes it impossible to delete files from windows explorer for example. If you want to fix this you basically have to bypass it by using "\\?\" on the front of the full path. The situation gets messy when you're trying to write an installer with node packages especially. https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- dom0 10y ago
  That's an issue that lies in the unfortunate intersection of 16/32 bit Windows, Windows NT and MS libc (MSVCRT), which support some combination of two and a half system designs.
  - frik 10y ago
    Microsoft could have fixed it with Win 64bit (Win64). They had done such a step before with the switch from Win16 to Win32 and the help of Win32s. With Win32 they cleaned up the old API, fixed things yet kept it source compatible when possible. Microsoft could have fixed so many things with Win64 starting with Windows 2003 64-bit. But no, Microsoft invested little on native Windows API between 2002 and 2012 - Longhorn (later Vista) and dotNet were the latest hype.
    - problems 10y ago
      Seems like they're just starting to fix it now with Win10 and only on applications that ship a special manifest (or on entire machines if you set a specific registry entry).
    - mikequinlan 10y ago
      Microsoft has partially fixed this in Windows 10 but it is not simple or easy. https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-4-6-2-and-long-paths-on-windows-10/ https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-...
recentdarkness 10y ago
And to add to the confusion unix domain sockets have a maximal length of something between 92 a 108. That's an implementation detail of platform it's running on. This in particular has been biting me already.
leni536 10y ago
I played around with glibc's getcwd() some time ago. With strace one can easily see how getcwd() works. If the current path is larger than PATH_MAX then the getcwd syscall fails. Then as I recall glibc uses '..'s recursively so it never has to call a syscall with a long relative path. If there is a non-user-readable directory in the path then the fallback method fails but the getcwd syscall works if the path is short enough. Bash also "cheats" by caching the working directory and updating it on 'cd' commands.
TheAceOfHearts 10y ago
This post got me curious, so I did a quick search on macOS 10.12. I found the values defined in "/System/Library/Frameworks/Kernel.framework/Headers/sys/syslimits.h". PATH_MAX is 1024, and NAME_MAX is 255. There's also an amusing todo question that looks like it might've been there for at least close to 20 years now: #define OPEN_MAX 10240 /* max open files per process - todo, make a config option? */
Animats 10y ago
All the UNIX/Linux/POSIX functions which take output "char *" params without a length should have been moved to deprecated header files a long time ago. Like 1990 or so. It's not too late.
- Tempest1981 10y ago
  Sometimes I dream of a world where C had a built-in string type. Imagine how much time could have been saved, and how many crashes prevented.
teddyh 10y ago
> This constant [PATH_MAX] is defined by POSIX… Well, it’s allowed by POSIX. A POSIX compatible system doesn’t have to define it if it has no such inherent restriction on path lengths. Indeed, the GNU Hurd does not have such a restriction, and consequently does not define it. This leads to many porting adventures for those trying to compile a program on GNU Hurd, believing their source code to be correct for any POSIX-compliant system.
manwe150 10y ago
If I understand the conclusion of the article right, it's that we should actually just use PATH_MAX? In particular, he points to the glibc implementation of realpath as being very correct. But it (like the man page description of it says), appears to prefer to use the hard-coded value of PATH_MAX, unless that value is unavailable and it is forced to query for the kernel _PC_PATH_MAX value instead. That's not what I would have expected. Did I miss something obvious?

67 points by eklitzke 10y ago

dom0 10y ago

Contrary to some previous entries [1][2] by The Author this one is accurate. [1] https://news.ycombinator.com/item?id=13896675 https://news.ycombinator.com/item?id=13896675 [2] https://news.ycombinator.com/item?id=13935256 https://news.ycombinator.com/item?id=13935256

derefr 10y ago

> The problem is that you can’t meaningfully define a constant like this in a header file. The maximum path size is actually to be something like a filesystem limitation, or at the very least a kernel parameter. AFAIK, "paths" aren't a thing filesystems think about. As far as a filesystem driver is concerned, paths are either—for open/read/write/etc.—plain inodes (a uint64), or—for directory-manipulation calls—an inode plus a ptrdiff_t to index into the dirent list†. The only things that care about NAME_MAX are lookup(2) [get inode given {inode, dirent}], and link(2) [put inode in {inode, dirent}]. So it's really only the kernel, through its syscall interface, that cares about paths—and so PATH_MAX is just a representation of the maximum size of a path the kernel is willing to accept in those syscalls. As if they each had a statically-allocated path[PATH_MAX] buffer your path got copied into. † Writing a FUSE filesystem is a great way to learn about what the kernel thinks a filesystem is. It's very different from the userland perspective. For example, from a filesystem driver's perspective, "file descriptors" don't exist! Instead, read(2) and write(2) calls are stateless, each call getting passed a kernel-side io-handle struct that must get re-evaluated for matching permissions on each IO operation. (You can do some up-front evaluation during the open(2) call, but, given that a file's permissions might change while you have a descriptor open to it, there's not much point.)

codys 10y ago
One of the things filesystems provide is a mapping from paths to data (files, directories, etc). How they provide this varies (tree of path elements, etc), but saying they don't think about paths (and implying they only think about path elements) is misleading. Nothing prevents a file system from simply putting all it's path to data mappings into a hash table (except the slight difficulty in implementing some common filesystem operations).
- derefr 10y ago
  Maybe on different OSes, but like I said above, on Linux, many operations will literally just pass a filesystem driver an inode. Your hypothetical driver would need to keep a double-mapping: from inodes to paths, and then from paths to whatever else. At which point, why bother with the second mapping? It's just slowing things down.
  - codys 10y ago
    The typical (generalized) setup is paths -> inodes (called "refers to data" or ""something"" in my other comments to avoid using the weird terminology we've somehow adapted for naming file system things) and inodes -> actual data on the disk. Whether you actually have that setup on disk is up to the fs format, but regardless of the format it is typically useful (and as you note, generic file system interfaces like the one in linux do require this) to have this extra indirection when tracking things on the software side (ie: not necessarily the storage side, though many file systems do have this type of indirection because it is also useful on the storage side). > Your hypothetical driver would need to keep a double-mapping: from inodes to paths, and then from paths to whatever else. As I've noted in another comment, inodes (`struct inode` in linux) typically contain a lot of information. No additional mapping would be required.
    - derefr 10y ago
      My point was that paths are part of Linux VFS, and that lookup is already done and invisible when the calls reach the FS driver. So there's no point in implementing an FS driver in terms of paths. It's redundant working-backward from something that was already done for you.
  - dom0 10y ago
    > Maybe on different OSes, Perhaps on a mainframe OS or something that doesn't have a VFS, but all VFS pretty much work the same way conceptually (BSD/nix, Linux, Windows).
- loeg 10y ago
  That would make for some very expensive lookup operations on subdirectories (you have to scan the entire hash table) and would entirely preclude hardlinks. So yeah, usually people do not write byzantine filesystems :-).
jstimpfle 10y ago
> For example, from a filesystem driver's perspective, "file descriptors" don't exist! Instead, read(2) and write(2) calls are stateless That seems hard to believe. It would inefficient to look for a file for each e.g. write(). I guess a filesystem defines its own implementation-defined open-file handle and the kernel translates that into an (integer) FD. > each call getting passed a kernel-side io-handle struct that must get re-evaluated for matching permissions on each IO operation. (You can do some up-front evaluation during the open(2) call, but, given that a file's permissions might change while you have a descriptor open to it, there's not much point.) That's not how it works. File modes are checked when you open the file. Once you have an open file (a file handle), the file modes no longer matter. File modes are more a "PATH" thing (there aren't per-filesystem variations and I assume permission checking is not done in file system code), although the file system must allocate the space to save the file mode bits and implement the VFS API.
- codys 10y ago
  Yep, this is misleading. "file descriptors" taken _literally_ don't exist (they are just the interface exposed to user space programs). Linux uses `struct file` internally as the generic file tracking, and multiple file descriptors can refer to a single `struct file`. `struct file` in turn refers to a `struct inode`, which represents an actual "something" on disk. Each file system driver stores a bunch of info in the `struct inode`. In addition to that tracking, there is generally a bunch of fs-specific data structures to handle lookup of names (in the typical case, these are essentially cached versions of what is stored on disk).
- loeg 10y ago
  > That seems hard to believe. It would inefficient to look for a file for each e.g. write(). I guess a filesystem defines its own implementation-defined open-file handle and the kernel translates that into an (integer) FD. At least in BSD, the syscall layer calls into the kernel virtual filesystem, which translates file descriptors into file objects. File objects have fo_read/fo_write methods implemented by different kinds of file object. Files backed by filesystems have an associated "vnode" (virtual inode object). Then the fo_read/fo_write layer invokes the VOP_READ or VOP_WRITE (for example) method on the vnode. VOP ("vnode operation") methods are implemented by individual filesystems. Multiple user-level file descriptors can refer to a single vnode. The filesystem's inode object hangs off a pointer from the generic vnode. Vnodes are unique (1:1) per real file (inode). So: read(fd, ...) -> sys_read -> kern_readv -> fget() (translate fd number into struct file object; check that the descriptor was opened read/write as appropriate for the operation) dofileread() -> fo_read() -> vn_io_fault() (fo_read method for vnodes) -> vn_read -> VOP_READ(vnode, uio, ioflags, ucred) (vnode for the file, an io descriptor ("uio"), io flags, and user credentials) So yeah, at the filesystem layet (VOP_READ), you no longer have the fd (or even the file object) in the BSD model. Linux's model is similar, but not completely identical. They call "vnodes" "inodes" instead, and may not have the intermediary file layer.

josteink 10y ago

That's tricky indeed. And doing things properly seems quite involved. For now I'll keep my limits.h. At least until I get a real-world bug-reports telling me this is causing real-world issues :)

loeg 10y ago

The GNU Hurd approach to PATH_MAX is to set it to something ridiculous like SIZE_MAX, something that cannot possible be allocated, to illustrate to programmers that it is a fiction. I don't think that's necessarily the best approach, but it matches reality more closely than typical Linux/BSD values (1024 or 4096).

the_mitsuhiko 10y ago
That seems like a bad idea because people will then use inconsistent max path lengths.
- jerf 10y ago
  What is the correct consistent max path length?
  - jstimpfle 10y ago
    It simplifies implementations. For example, typically you need to be able to store paths in a consecutive area of memory or storage. Knowing that paths don't exceed a certain size means you can know that a particular allocation strategy (like static allocation, for a simple example) will be sufficient.
    - klodolph 10y ago
      Mostly it's for stuff like readlink(), which returns a path in a user-supplied buffer. How big should the buffer be? The Linux man page actually suggests doing lstat() to size the buffer, with a note that this is susceptible to a race condition!
      - dom0 10y ago
        readlink(2) takes a buffer and the buffer's size, so it's safe. You just enter a race-controlled loop, as always.

jstimpfle 10y ago

The essence of the article: PATH_MAX applies to the syscall interface. It's not related to file systems. Paths aren't a file system thing, but simply a convenient means of addressing files. Basically they are URLs for local resources. And that totally makes sense once you understand that files are basically "objects" (in the OO sense) identified by inodes instead of memory addresses. A file system implements the graph of these objects (linked by special file objects called directories). The fact that one can cross file system boundaries using file paths also indicates that file paths are none of a file system's business.

masklinn 10y ago
> It's not related to file systems. Paths aren't a file system thing, but simply a convenient means of addressing files. > The fact that one can cross file system boundaries using file paths also indicates that file paths are none of a file system's business. The filesystem knows about file names, stores them, and puts limits on them (often 255 code units though some are lower — FAT16's 8.3, HFS's 31 — and some are higher — Reiser4's 3976 bytes). A file path is nothing but a concatenation of a bunch of file names and separators ergo file paths are, in fact, an FS's business. And while that's mostly fallen out of style there are still length-limited-path filesystems: ISO-9660 and UDF for instance.
- jstimpfle 10y ago
  > A file path is nothing but a concatenation of a bunch of file names and separators ergo file paths are, in fact, an FS's business. This is a non sequitur.

problems 10y ago

Windows has a similarly disastrous situation where most tools and APIs follow MAX_PATH - which is defined to be 260 chars. But that doesn't affect the actual filesystem or syscall interface, just common APIs and tools. This makes it impossible to delete files from windows explorer for example. If you want to fix this you basically have to bypass it by using "\\?\" on the front of the full path. The situation gets messy when you're trying to write an installer with node packages especially. https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

dom0 10y ago
That's an issue that lies in the unfortunate intersection of 16/32 bit Windows, Windows NT and MS libc (MSVCRT), which support some combination of two and a half system designs.
- frik 10y ago
  Microsoft could have fixed it with Win 64bit (Win64). They had done such a step before with the switch from Win16 to Win32 and the help of Win32s. With Win32 they cleaned up the old API, fixed things yet kept it source compatible when possible. Microsoft could have fixed so many things with Win64 starting with Windows 2003 64-bit. But no, Microsoft invested little on native Windows API between 2002 and 2012 - Longhorn (later Vista) and dotNet were the latest hype.
  - problems 10y ago
    Seems like they're just starting to fix it now with Win10 and only on applications that ship a special manifest (or on entire machines if you set a specific registry entry).
  - mikequinlan 10y ago
    Microsoft has partially fixed this in Windows 10 but it is not simple or easy. https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-4-6-2-and-long-paths-on-windows-10/ https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-...

recentdarkness 10y ago

And to add to the confusion unix domain sockets have a maximal length of something between 92 a 108. That's an implementation detail of platform it's running on. This in particular has been biting me already.

leni536 10y ago

I played around with glibc's getcwd() some time ago. With strace one can easily see how getcwd() works. If the current path is larger than PATH_MAX then the getcwd syscall fails. Then as I recall glibc uses '..'s recursively so it never has to call a syscall with a long relative path. If there is a non-user-readable directory in the path then the fallback method fails but the getcwd syscall works if the path is short enough. Bash also "cheats" by caching the working directory and updating it on 'cd' commands.

TheAceOfHearts 10y ago

This post got me curious, so I did a quick search on macOS 10.12. I found the values defined in "/System/Library/Frameworks/Kernel.framework/Headers/sys/syslimits.h". PATH_MAX is 1024, and NAME_MAX is 255. There's also an amusing todo question that looks like it might've been there for at least close to 20 years now: #define OPEN_MAX 10240 /* max open files per process - todo, make a config option? */

Animats 10y ago

All the UNIX/Linux/POSIX functions which take output "char *" params without a length should have been moved to deprecated header files a long time ago. Like 1990 or so. It's not too late.

Tempest1981 10y ago
Sometimes I dream of a world where C had a built-in string type. Imagine how much time could have been saved, and how many crashes prevented.

teddyh 10y ago

> This constant [PATH_MAX] is defined by POSIX… Well, it’s allowed by POSIX. A POSIX compatible system doesn’t have to define it if it has no such inherent restriction on path lengths. Indeed, the GNU Hurd does not have such a restriction, and consequently does not define it. This leads to many porting adventures for those trying to compile a program on GNU Hurd, believing their source code to be correct for any POSIX-compliant system.

manwe150 10y ago

If I understand the conclusion of the article right, it's that we should actually just use PATH_MAX? In particular, he points to the glibc implementation of realpath as being very correct. But it (like the man page description of it says), appears to prefer to use the hard-coded value of PATH_MAX, unless that value is unavailable and it is forced to query for the kernel _PC_PATH_MAX value instead. That's not what I would have expected. Did I miss something obvious?