5 ms·

These problems with path names on the Mac have been an issue ever since FolderShare was originally ported to Mac OS X. Those limits are straight out of the Win

These problems with path names on the Mac have been an issue ever since FolderShare was originally ported to Mac OS X. Those limits are straight out of the Windows API (http://msdn.microsoft.com/en-us/library/aa365247.aspx http://msdn.microsoft.com/en-us/library/aa365247.aspx).

asveikau 12y ago
You can bypass most of these limitations on Windows by prefixing your path with \\?\. Suddenly instead of 260 chars you get 32k if memory serves. And trailing dots and spaces (at the end of a file or before its extension -- limits that come straight out of FAT's direntry structure) are OK. So really I'd say these are limits of not knowing or bothering to use the escape hatch in Windows's support for this stuff.
- 13 12y ago
  I always wondered why Wine prefixed unixy paths with that string.
- xenophonf 12y ago
  I completely agree. Because of this we ran into all sorts of problems getting FolderShare/Live Mesh/whatever Microsoft calls it these days to sync files between Macs and PCs. I ended up writing a Perl script that would periodically scan the Mac's shared folder for non-ASCII file names and normalize them by brute force. It was ugly. I told my boss that if it broke, he got to keep both pieces.
- Someone1234 12y ago
  Unfortunately (or fortunately depending on your perspective) Microsoft takes a very "lowest common denominator" approach to backwards compatibility. A LOT of third party software would definitely be broken if they altered Windows filename restrictions. They would also have to drop support for FAT32 as you pointed out. Right now a lot of Windows 95 software continues to run 20 years later. And in no small part because of this type of attitude and all the runtime patching Microsoft does.
  - ethomson 12y ago
    They did alter the Windows filename restrictions. You can happily create a file that is 1000 characters long, and your Windows 95 software will likely refuse to open it. (Hell, Windows Explorer refuses to deal with it.) Now we've got a situation where we keep crowing about how changing MAX_PATH would break compatibility. Except, of course, that we already did that by adding the stupid '\\?\' APIs (which are themselves a mess). So now we don't actually have backward compatibility for those programs (we just make it so hard to use long paths that we virtually have backward compatibility.) And we have file paths that resemble 1995. We don't have the lowest common denominator, we just have the worst of both worlds.
    - Someone1234 12y ago
      Fair enough. I'm surprised with Metro/Modern/RT/whatever's APIs they didn't try and move forward by utilising \\?\ in Windows.Storage.*. But from what I've been able to tell they have the same limitations as other .Net or Win32 APIs.
  - asveikau 12y ago
    > They would also have to drop support for FAT32 as you pointed out. This is a little dubious. When you load a FAT driver on a *nix system, you get the limitations of FAT. Likewise there are lots of fancy filesystem APIs in Windows that fail on a FAT volume and work on NTFS. That said, there are a lot of quirky NTFS behaviors that are ultimately rooted in FAT. I used to get frustrated by a lot of them, then I started writing a FAT driver just for fun ... and it made sense. All the goofy things about naming start to make sense if the direntry "is your inode" and the extra space in the name is padded with spaces...
- AlyssaRowan 12y ago
  Oh, yes, those. Please be advised that they are possibly one of the ugliest bits of the Windows API. You could blog about this nightmare. Particular hairballs to watch out for regarding the path length/filename length issue (please forgive me if I've got these slightly wrong as I'm paraphrasing roughly from memory): • Normal MAX_PATH == 260 WCHARs (hardcoded in the Windows API), including the drive letter, colon, backslash, and the trailing NUL; • Directory names in the normal API can't be longer than (MAX_PATH-12) WCHARs, so that they can always contain a FILENAME.EXT; • You can use extended-length paths with a "\\?\" prefix (exactly); UNC network drive paths which you want to be extended-length should start "\\?\UNC\" (exactly), then append the server name, backslash, share name, backslash, path and file etc; • Extended-length paths only generally work with W/Ex APIs; • The buffer used for extended-length paths is 65536 bytes, including the trailing NUL, but that also needs to contain any expansions during processing; • Names passed with the extended-length API do not have to be valid or normalised Unicode; • Components (split between backslashes, i.e. filenames) of an extended-length path are in fact still limited to GetVolumeInformation/GetVolumeInformationByHandleW/etc's lpMaximumComponentLength, which is often 255 (WCHARs) but not always; • It counts WCHARs not codepoints (look out for codepoints beyond the Basic Multilingual Plane!); • There are no relative paths or current directories in this API, everything is absolute, so no plain filenames, and "." and ".." are actually valid names not directory references; • Most of the Explorer shell, and all of cmd.exe (as of W8, don't know about W10) doesn't use them, and because of the above you can't use them as a cd, you can pushd them but stuff will break; • Several APIs don't even support it; • So you can make files and directories which a good half of Windows and related applications can't do anything with, and are very pesky to delete… …and beyond that is where it gets really hairy. The code in this area involving devices (from "\\.\PhysicalDisk0" to good ol' "COM1") is particularly Cthulhu-esque, as is the code regarding Terminal Services and Hyper-V. (A good sweep through there looking for off-by-ones might be fruitful, as those with sharp eyes may have spotted from the above.) I'll make allowances for needing to be incredibly careful about backwards compatibility to a really old VMS/GEMDOSish codebase, and being patch-upon-patch-upon-patch as a result. But that they had the opportunity to fix it in NT, and sadly missed it irks me: that and the locking semantics (especially regarding running executables) are probably my least favourite characteristics of Windows. Linux, for example, is much easier about this particular kind of thing, but its flexibility does have its own problems when absolutely anything can be in a filename and people often forget that (for example, the delightful fun that can ensue when someone starts a filename with "-", say, "-rf" and someone does a "rm " which globs it first, because globbing doesn't know or care what switches are, but you* will).

asveikau 12y ago

You can bypass most of these limitations on Windows by prefixing your path with \\?\. Suddenly instead of 260 chars you get 32k if memory serves. And trailing dots and spaces (at the end of a file or before its extension -- limits that come straight out of FAT's direntry structure) are OK. So really I'd say these are limits of not knowing or bothering to use the escape hatch in Windows's support for this stuff.

13 12y ago
I always wondered why Wine prefixed unixy paths with that string.
xenophonf 12y ago
I completely agree. Because of this we ran into all sorts of problems getting FolderShare/Live Mesh/whatever Microsoft calls it these days to sync files between Macs and PCs. I ended up writing a Perl script that would periodically scan the Mac's shared folder for non-ASCII file names and normalize them by brute force. It was ugly. I told my boss that if it broke, he got to keep both pieces.
Someone1234 12y ago
Unfortunately (or fortunately depending on your perspective) Microsoft takes a very "lowest common denominator" approach to backwards compatibility. A LOT of third party software would definitely be broken if they altered Windows filename restrictions. They would also have to drop support for FAT32 as you pointed out. Right now a lot of Windows 95 software continues to run 20 years later. And in no small part because of this type of attitude and all the runtime patching Microsoft does.
- ethomson 12y ago
  They did alter the Windows filename restrictions. You can happily create a file that is 1000 characters long, and your Windows 95 software will likely refuse to open it. (Hell, Windows Explorer refuses to deal with it.) Now we've got a situation where we keep crowing about how changing MAX_PATH would break compatibility. Except, of course, that we already did that by adding the stupid '\\?\' APIs (which are themselves a mess). So now we don't actually have backward compatibility for those programs (we just make it so hard to use long paths that we virtually have backward compatibility.) And we have file paths that resemble 1995. We don't have the lowest common denominator, we just have the worst of both worlds.
  - Someone1234 12y ago
    Fair enough. I'm surprised with Metro/Modern/RT/whatever's APIs they didn't try and move forward by utilising \\?\ in Windows.Storage.*. But from what I've been able to tell they have the same limitations as other .Net or Win32 APIs.
- asveikau 12y ago
  > They would also have to drop support for FAT32 as you pointed out. This is a little dubious. When you load a FAT driver on a *nix system, you get the limitations of FAT. Likewise there are lots of fancy filesystem APIs in Windows that fail on a FAT volume and work on NTFS. That said, there are a lot of quirky NTFS behaviors that are ultimately rooted in FAT. I used to get frustrated by a lot of them, then I started writing a FAT driver just for fun ... and it made sense. All the goofy things about naming start to make sense if the direntry "is your inode" and the extra space in the name is padded with spaces...
AlyssaRowan 12y ago
Oh, yes, those. Please be advised that they are possibly one of the ugliest bits of the Windows API. You could blog about this nightmare. Particular hairballs to watch out for regarding the path length/filename length issue (please forgive me if I've got these slightly wrong as I'm paraphrasing roughly from memory): • Normal MAX_PATH == 260 WCHARs (hardcoded in the Windows API), including the drive letter, colon, backslash, and the trailing NUL; • Directory names in the normal API can't be longer than (MAX_PATH-12) WCHARs, so that they can always contain a FILENAME.EXT; • You can use extended-length paths with a "\\?\" prefix (exactly); UNC network drive paths which you want to be extended-length should start "\\?\UNC\" (exactly), then append the server name, backslash, share name, backslash, path and file etc; • Extended-length paths only generally work with W/Ex APIs; • The buffer used for extended-length paths is 65536 bytes, including the trailing NUL, but that also needs to contain any expansions during processing; • Names passed with the extended-length API do not have to be valid or normalised Unicode; • Components (split between backslashes, i.e. filenames) of an extended-length path are in fact still limited to GetVolumeInformation/GetVolumeInformationByHandleW/etc's lpMaximumComponentLength, which is often 255 (WCHARs) but not always; • It counts WCHARs not codepoints (look out for codepoints beyond the Basic Multilingual Plane!); • There are no relative paths or current directories in this API, everything is absolute, so no plain filenames, and "." and ".." are actually valid names not directory references; • Most of the Explorer shell, and all of cmd.exe (as of W8, don't know about W10) doesn't use them, and because of the above you can't use them as a cd, you can pushd them but stuff will break; • Several APIs don't even support it; • So you can make files and directories which a good half of Windows and related applications can't do anything with, and are very pesky to delete… …and beyond that is where it gets really hairy. The code in this area involving devices (from "\\.\PhysicalDisk0" to good ol' "COM1") is particularly Cthulhu-esque, as is the code regarding Terminal Services and Hyper-V. (A good sweep through there looking for off-by-ones might be fruitful, as those with sharp eyes may have spotted from the above.) I'll make allowances for needing to be incredibly careful about backwards compatibility to a really old VMS/GEMDOSish codebase, and being patch-upon-patch-upon-patch as a result. But that they had the opportunity to fix it in NT, and sadly missed it irks me: that and the locking semantics (especially regarding running executables) are probably my least favourite characteristics of Windows. Linux, for example, is much easier about this particular kind of thing, but its flexibility does have its own problems when absolutely anything can be in a filename and people often forget that (for example, the delightful fun that can ensue when someone starts a filename with "-", say, "-rf" and someone does a "rm " which globs it first, because globbing doesn't know or care what switches are, but you* will).