8 ms·

The Definitive Guide on Win32 to NT Path Conversion (2016)

113 points by camtarn 7y ago

gumby 7y ago
Shoutout for calling himself a "path'ological reverse engineer"
deleted 7y ago
[deleted]
maxxxxx 7y ago
Windows paths can be really infuriating. We constantly run into problems with the 255 file name length limit and even if you use the APIs to create longer paths then Explorer or other apps can't deal with these names. It's hard to believe that after all these years this is still a problem but I guess a clunky foundation doesn't prevent one from being successful.
saagarjha 7y ago
> we can only store at most 215-1 characters, which just happens to be 32,767 I think the formatting ate the superscript here.
- hoseja 7y ago
  <span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2</span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15</span> Yeah, I think it did.
Endy 7y ago
And yet with all this, I still prefer Win16/Win32 paths and system layout as they are implemented for user needs compared to any UNIX/BSD/Linux pathing and system structure.
- m45t3r 7y ago
  Windows paths is full of WTFs: - The main driver is named C:\ because DOS used A:\ and B:\ for floppy disks! C'mon, floppy disks were not used as the main disk since god knows why (I think even MS-DOS itself was already supposed to run from HDDs and not floppy disks). - I can't have a file named LPT1, COM, AUX, PRN, NUL, etc. This would be fine, however I can't even have a file named aux.ext, for example. I know this is for compatibility purposes with DOS, however the fact that the error message is terrible does not help. -255 char limit. I know that you can use an API that allows more characters than this, however this does not work everywhere (last time I remember having problems to access those files in Windows Explorer; I think they fixed this in the latest versions of Windows 10, though). - Some common characters are prohibit in filenames, like ? or :. Unix has some limitations too, however the Windows are much more ridiculous. - In 64 bits systems, C:\Windows\SysWoW64\system32 is for 32 bits DLLs, while C:\Windows\system32 is for 64 bits DLLs. I really is curious why Microsoft choose this one, it is not like they needed for legacy software [1] (since 64 bits were new systems not old ones). I mean, most of the Windows WTFs are for compatibility purposes, however Unix has a much more sensible path rules even if you consider that Unix has much more history than Windows. There is much less WTFs in Unix paths (like /usr/bin vs /bin, or /opt vs. /usr/local). I can't really understand what is going on with Windows paths. Unix paths are far more sane to me IMO. [1]: If someone can link to me the reason why I would be thankful. I know that Microsoft engineers are not stupid, I just find it really curious why C:\Windows\system64 was not used, for example.
  - twall 7y ago
    [1] was also all about compatibility. It's quite amazing the lengths Microsoft went through to preserve compatibility during the 32 to 64 bit transition, and how well it continues to work to this day. That all came at the expense of confusing the hell out of mere users or developers looking back years later. It was important that programs compiled for a 32-bit windows could run without modifications on a 64-bit windows system. While some amount of 32-bit application recompiles or rewrites was inevitable, a surprising number of them could get by without modifications thanks to Microsoft's emulation & compatibility efforts. "wow64", or windows on window64, is an emulator of sorts to run 32 bit programs in a 64-bit windows environment with various behavior changes designed to trick 32-bit applications into thinking they were really running on a 32-bit system. The syswow64\system32 thing is one such effort. If I have a 32-bit program running I may have hard coded paths to files or libraries my program needs. If my program does a fopen("C:\Windows\System32\whatever", mode) expecting to find a dll and it's not there or cannot be opened because that's now where 64-bit dlls live, my program will blow up. So, when running in wow64 mode, the path "C:\Windows\System32" resolves to "C:\Windows\SysWow64\System32" when running a 32-bit program on 64-bit Windows. When running a 64-bit program, it resolves to what you'd "expect", "C\Windows\System32". That's a strange name for a 64-bit system folder, as you note, but imagine if it were something like system64 -- then if I want to support both 32 & 64 bit systems, I'd need different paths and logic in my program (and it's installer, and in all my tests, and ...) to switch between the two areas for 32-bit and 64-bit systems. By having the folder use the same name (from the program's perspective) everything gets a lot simpler. This same approach happens when you are looking at Program Files vs Program Files (x86), and in the registry, where HKLM\SOFTWARE\ is 64-bit, while HKLM\SOFTWARE\WOW6432Node\ is for 32-on-64. This emulation is so complete that it is actually difficult for a 32-bit program with awareness of Microsoft's emulation to actually find the files in SysWow64\System32. If for example, I had an installer program that wants to install my software for both kinds of system, it is difficult if you don't follow best practices and use 2 MSI installers for each bitness and wrap them in a bootstrapping installer program. I learned all this the hard way, by shipping a 32 & 64 bit windows library. We offer both 32 & 64 bit versions because it has to work with old 32-bit software, even to this day. It is quite confusing but if you squint hard, understand their reasoning and follow it through all the use cases it makes a decent amount of sense.
voldacar 7y ago
The ugliness of the windows API never ceases to amaze me. Functions with names like RtlDetermineDosPathNameType_U and return types like PCWSTR just make my eyes bleed.
- colatkinson 7y ago
  My personal favorite is PCZZWSTR: a const pointer to a contiguous WCHAR array, where each substring is NUL-terminated, and the array itself is also terminated by a NUL. Thus (I assume) the ZZ--zero zero. I've only ever seen it used in the SHFILEOPSTRUCTW struct.
  - wtallis 7y ago
    That reads like one of the instruction mnemonics from https://twitter.com/x86instructions https://twitter.com/x86instructions
  - voldacar 7y ago
    >SHFILEOPSTRUCTW now that takes the cake
    - whoopdedo 7y ago
      Nah. That would be INITCOMMONCONTROLSEX. And the related LVM_SORTITEMSEX, DDSCAPSEX, and of course MPR_CREDENTIALSEX_0.
      - saagarjha 7y ago
        With Microsoft's use of ALLCAPS and using an "Ex" prefix, this was all but guaranteed…
    - msclrhd 7y ago
      How about SHITEMID in the shlwapi DLL? That's supposed to be SH-ITEM-ID.
    - Barrin92 7y ago
yokohummer7 7y ago
> Somewhere in Explorer something is passing one of these shell names to a file API, which is interpreting it as a Drive Relative path for the ‘:’ drive. Isn't this just a pure bug? I'm quite surprised such an intimidating behavior still exists in today's Windows...
MarekKnapek 7y ago
> Fortunately this doesn’t seem to work in the Explorer shell, but it does work when passed to native APIs such as CreateFile. CreateFile is Win32 API, not Native API. NtCreateFile would be Native API.
makomk 7y ago
Gotta love how there's a whole chunk of complicated compatibility code to emulate the old DOS-ism of having per-drive current directories.

113 points by camtarn 7y ago

gumby 7y ago

Shoutout for calling himself a "path'ological reverse engineer"

deleted 7y ago

[deleted]

maxxxxx 7y ago

Windows paths can be really infuriating. We constantly run into problems with the 255 file name length limit and even if you use the APIs to create longer paths then Explorer or other apps can't deal with these names. It's hard to believe that after all these years this is still a problem but I guess a clunky foundation doesn't prevent one from being successful.

saagarjha 7y ago

> we can only store at most 215-1 characters, which just happens to be 32,767 I think the formatting ate the superscript here.

hoseja 7y ago
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2</span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15</span> Yeah, I think it did.

Endy 7y ago

And yet with all this, I still prefer Win16/Win32 paths and system layout as they are implemented for user needs compared to any UNIX/BSD/Linux pathing and system structure.

m45t3r 7y ago
Windows paths is full of WTFs: - The main driver is named C:\ because DOS used A:\ and B:\ for floppy disks! C'mon, floppy disks were not used as the main disk since god knows why (I think even MS-DOS itself was already supposed to run from HDDs and not floppy disks). - I can't have a file named LPT1, COM, AUX, PRN, NUL, etc. This would be fine, however I can't even have a file named aux.ext, for example. I know this is for compatibility purposes with DOS, however the fact that the error message is terrible does not help. -255 char limit. I know that you can use an API that allows more characters than this, however this does not work everywhere (last time I remember having problems to access those files in Windows Explorer; I think they fixed this in the latest versions of Windows 10, though). - Some common characters are prohibit in filenames, like ? or :. Unix has some limitations too, however the Windows are much more ridiculous. - In 64 bits systems, C:\Windows\SysWoW64\system32 is for 32 bits DLLs, while C:\Windows\system32 is for 64 bits DLLs. I really is curious why Microsoft choose this one, it is not like they needed for legacy software [1] (since 64 bits were new systems not old ones). I mean, most of the Windows WTFs are for compatibility purposes, however Unix has a much more sensible path rules even if you consider that Unix has much more history than Windows. There is much less WTFs in Unix paths (like /usr/bin vs /bin, or /opt vs. /usr/local). I can't really understand what is going on with Windows paths. Unix paths are far more sane to me IMO. [1]: If someone can link to me the reason why I would be thankful. I know that Microsoft engineers are not stupid, I just find it really curious why C:\Windows\system64 was not used, for example.
- twall 7y ago
  [1] was also all about compatibility. It's quite amazing the lengths Microsoft went through to preserve compatibility during the 32 to 64 bit transition, and how well it continues to work to this day. That all came at the expense of confusing the hell out of mere users or developers looking back years later. It was important that programs compiled for a 32-bit windows could run without modifications on a 64-bit windows system. While some amount of 32-bit application recompiles or rewrites was inevitable, a surprising number of them could get by without modifications thanks to Microsoft's emulation & compatibility efforts. "wow64", or windows on window64, is an emulator of sorts to run 32 bit programs in a 64-bit windows environment with various behavior changes designed to trick 32-bit applications into thinking they were really running on a 32-bit system. The syswow64\system32 thing is one such effort. If I have a 32-bit program running I may have hard coded paths to files or libraries my program needs. If my program does a fopen("C:\Windows\System32\whatever", mode) expecting to find a dll and it's not there or cannot be opened because that's now where 64-bit dlls live, my program will blow up. So, when running in wow64 mode, the path "C:\Windows\System32" resolves to "C:\Windows\SysWow64\System32" when running a 32-bit program on 64-bit Windows. When running a 64-bit program, it resolves to what you'd "expect", "C\Windows\System32". That's a strange name for a 64-bit system folder, as you note, but imagine if it were something like system64 -- then if I want to support both 32 & 64 bit systems, I'd need different paths and logic in my program (and it's installer, and in all my tests, and ...) to switch between the two areas for 32-bit and 64-bit systems. By having the folder use the same name (from the program's perspective) everything gets a lot simpler. This same approach happens when you are looking at Program Files vs Program Files (x86), and in the registry, where HKLM\SOFTWARE\ is 64-bit, while HKLM\SOFTWARE\WOW6432Node\ is for 32-on-64. This emulation is so complete that it is actually difficult for a 32-bit program with awareness of Microsoft's emulation to actually find the files in SysWow64\System32. If for example, I had an installer program that wants to install my software for both kinds of system, it is difficult if you don't follow best practices and use 2 MSI installers for each bitness and wrap them in a bootstrapping installer program. I learned all this the hard way, by shipping a 32 & 64 bit windows library. We offer both 32 & 64 bit versions because it has to work with old 32-bit software, even to this day. It is quite confusing but if you squint hard, understand their reasoning and follow it through all the use cases it makes a decent amount of sense.

voldacar 7y ago

The ugliness of the windows API never ceases to amaze me. Functions with names like RtlDetermineDosPathNameType_U and return types like PCWSTR just make my eyes bleed.

colatkinson 7y ago
My personal favorite is PCZZWSTR: a const pointer to a contiguous WCHAR array, where each substring is NUL-terminated, and the array itself is also terminated by a NUL. Thus (I assume) the ZZ--zero zero. I've only ever seen it used in the SHFILEOPSTRUCTW struct.
- wtallis 7y ago
  That reads like one of the instruction mnemonics from https://twitter.com/x86instructions https://twitter.com/x86instructions
- voldacar 7y ago
  >SHFILEOPSTRUCTW now that takes the cake
  - whoopdedo 7y ago
    Nah. That would be INITCOMMONCONTROLSEX. And the related LVM_SORTITEMSEX, DDSCAPSEX, and of course MPR_CREDENTIALSEX_0.
    - saagarjha 7y ago
      With Microsoft's use of ALLCAPS and using an "Ex" prefix, this was all but guaranteed…
  - msclrhd 7y ago
    How about SHITEMID in the shlwapi DLL? That's supposed to be SH-ITEM-ID.
  - Barrin92 7y ago

yokohummer7 7y ago

> Somewhere in Explorer something is passing one of these shell names to a file API, which is interpreting it as a Drive Relative path for the ‘:’ drive. Isn't this just a pure bug? I'm quite surprised such an intimidating behavior still exists in today's Windows...

MarekKnapek 7y ago

> Fortunately this doesn’t seem to work in the Explorer shell, but it does work when passed to native APIs such as CreateFile. CreateFile is Win32 API, not Native API. NtCreateFile would be Native API.

makomk 7y ago

Gotta love how there's a whole chunk of complicated compatibility code to emulate the old DOS-ism of having per-drive current directories.