5 ms·

The opt-out is to use the unicode windows APIs (the functions ending in "w" instead of "a"). This also magically fixes all issues with paths longer than 260 cha

The opt-out is to use the unicode windows APIs (the functions ending in "w" instead of "a"). This also magically fixes all issues with paths longer than 260 characters (if you add a "\\?\" prefix or set you manifest correctly), and has been available and recommended since Windows XP. I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000.

vessenes 2y ago
I think the issue is that native OS things like the windows command line, say, don’t always do this. Check the results of their ‘cd’ commands with Japanese Yen characters introduced. You can see that the path descriptor somehow has updated to a directory name with Yen (or a wide backslash) in it, while the file system underneath has munged, and put them into an actual directory. It’s precisely the problem that you can’t control the rest of the API surface to use W that is the source of the difficulties.
Sharlin 2y ago
As mentioned elsewhere in this discussion, 99% of the time the cause is likely the use of standard C functions (or C++ `std::string`…) instead of MS's nonstandard wide versions. Which of course is a ubiquitous practice in portable command-line software like curl.
- pishpash 2y ago
  So the culprit is still the software writer. They should have wrapped the C++ library for OS-specific behavior on Windows. Because they are publishing buggy software and calling it cross-platform.
  - bayindirh 2y ago
    curl first released in 1996, shortly after Windows 95 has born and runs on numerous Windows versions even today. So, how many different versions shall be maintained? Are you going to help one of these versions? On top of that, how many new gotchas these “modern” Windows functions hide, and how many fix cycles are required to polish them to the required level?
    - thrdbndndn 2y ago
      If we're talking about curl specifically, I absolutely think they would (NOT "should") fix/workaround it if there are actually common problems caused by it. Yes it would have required numerous fix cycles, but curl in my mind is such a polished product and they would have bit the bullet.
      - 8n4vidtmkvmk 2y ago
        Why would/should they? I've never paid for curl. Who even develops it? Sounds like a thankless job to fix obscure worstfit bugs.
        thrdbndndn 2y ago
        Why would they develop curl at all by your logic? They fix bugs because they simply want their product to be better, if if I were to take a guess? Like, I'm sure curl's contributors worked on OS-specific problems before, and it wouldn't be the last. > to fix obscure worstfit bugs. Again my premise is "if there are actually common problems caused by it". This specific bug does sound like that, at least not for now.
        bayindirh 2y ago
        > Why would/should they? Because they care. That's it. > I've never paid for curl. I'm sure people who develop it doesn't want money and fame, but they're just doing what they like. However, curl has commercial support contracts if you need. > Who even develops it? Daniel Stenberg et al. Daniel can be found at https://daniel.haxx.se https://daniel.haxx.se. > Sounds like a thankless job to fix obscure worstfit bugs. It may look thankless, but it's not. curl is critical infrastructure at this point. While https://xkcd.com/2347/ https://xkcd.com/2347/ applies squarely to cURL, it's actually nice that the lead developer is making some money out of his endeavor.
      - bayindirh 2y ago
        You're right, if the problems created by this are big enough, the team will fix them without any fanfare and whining. However, in neither case this is a shortcoming of curl. They'd be responding to a complicated problem caused by the platform they're running on.
  - deleted 2y ago
    [deleted]
  - account42 2y ago
    I'm sure those software writers will be happy to refund your purchase. It's not their fault that Microsofts standard C implementation is faulty.
- smatija 2y ago
  A lot of details is in linked curl hackerone: https://hackerone.com/reports/2550951 https://hackerone.com/reports/2550951
- account42 2y ago
  std::string is not an issue, how you get strings from the environment into it is. You can use W functions and convert the WTF-16 strings you get to WTF-8 and use that in std::string without problems.
p_ing 2y ago
> paths longer than 260 characters (if you add a "\\?\" prefix or set you manifest correctly) A long ago released build of Windows 10 did this automatically so no need for adjustments anymore, 32k is the max.... ...except for Office! It can't handle long paths. But Office has always been hacky (the title bar, for example).
comex 2y ago
_Or_ set your application to use UTF-8 for the "A" APIs. Apparently this is supported as of a Windows 10 update from 2019. [1] [1] https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...
- asveikau 2y ago
  It should have been supported approximately 20 years earlier than that. I was coding against Win32 looong before 2019 and wondering for years why they wouldn't let you. An explanation I heard ~10 years prior is that doing so exposed bugs in CRT and nobody wanted to fix them.
  - pjmlp 2y ago
    Not sure what that has to do with CRT, given that it isn't part of Win32.
    - garganzol 2y ago
      CRT in a form of msvcrt.dll file had a de-facto presence in Windows since the end of 1990s. Later on, since 2018 or so, CRT availability was formalized in Windows API in form of ucrtbase.dll module.
      - pjmlp 2y ago
        Which doesn't change the fact that Win32 doesn't depend on it.
        okanat 2y ago
        It is extremely hard to create an application that doesn't depend on CRT on Windows. CRT provides tables for handlers of SEH exceptions and provides the default event handlers. Win32 headers have hard dependencies of the handler tables CRT provides. So you need to go quite a bit out of your way to hack deep Win32 headers. Loading DLLs etc also may call CRT functions. You can read Mingw64 source to see how many hacks they had to do to make it work.
cesarb 2y ago
> I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000. Nowadays, it's either for historical reasons (code written back when supporting Windows 9x was important, or even code migrated from Windows 3.x), or out of a desire to support non-Windows systems. Most operating systems use a byte-based multi-byte encoding (nowadays usually UTF-8) as their native encoding, instead of UTF-16.
asveikau 2y ago
I share your recommendations of always using PWSTR when using windows apis. > I'm not sure why the non-unicode APIs are still so commonly used I think because the rest of the C world uses char* with utf-8, so that is what people are habituated to. Setting the ACP to CP_UTF8 would have solved a lot of problems, but I believe that's only been supported for a short period of time, bafflingly.
- account42 2y ago
  > Setting the ACP to CP_UTF8 would have solved a lot of problems, but I believe that's only been supported for a short period of time, bafflingly. It wouldn't solve all encoding problems though because most Windows APIs can store/return invalid UTF-16 which you can't represent in CP_UTF8 - you'd need a CP_WTF8 for that which doesn't even exist so you have to use the W APIs and do the conversion yourself.
ack_complete 2y ago
Using \\?\ has a downside: since it bypasses Win32's path processing, it also prevents relative paths like d:test.txt from working. Kind of annoying on the command line with tools like 7z.exe.
- account42 2y ago
  Sounds more like an upside TBH.
Thorrez 2y ago
>I'm not sure why the non-unicode APIs are still so commonly used. Even argv is affected on Windows. That's part of the C and C++ standard, not really a Windows API. Telling all C/C++ devs they need to stop using argv is kind of a tough ask.
shakna 2y ago
You also have to use wmain instead of main, with a wchar_t argv, otherwise the compiled-in argparser will be calling the ANSI version. In other words... Anyone using MSVC and the cross-platform standardised and normal C system, are hit by this. Oh, and wmain is a VisualC thing. It isn't found on other platforms. Not standardised.
- mort96 2y ago
  Writing cross platform code which consistently uses UCS-2 wchar_t* on Windows and UTF-8 char* on UNIX-like systems sounds like absolute hell
  - lmz 2y ago
    A wchar_t "native" libc implementation would be an interesting thing.
  - account42 2y ago
    It's not that bad really - you just convert at the win32 API call boundary. Also, it's not UCS-2. Also not UTF-16. Windows uses WTF-16 internally and if you want 100% compatibility that's what you need to target.
cryptonector 2y ago
The other opt-out might be to opt into UTF-8 support for the "A" functions.
cryptonector 2y ago
> I'm not sure why the non-unicode APIs are still so commonly used. Simple: portable code meant to run on Unix (where UTF-8 is king) and Windows -> want to use UTF-8 codepage on Windows and the "A" APIs.
account42 2y ago
> I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000. A lot of the uses are indirectly via standard C API functions that are effectively broken on Windows but work just well enough (i.e. work with ASCII) that their use goes unnoticed when someone ports something to Windows.

vessenes 2y ago

I think the issue is that native OS things like the windows command line, say, don’t always do this. Check the results of their ‘cd’ commands with Japanese Yen characters introduced. You can see that the path descriptor somehow has updated to a directory name with Yen (or a wide backslash) in it, while the file system underneath has munged, and put them into an actual directory. It’s precisely the problem that you can’t control the rest of the API surface to use W that is the source of the difficulties.

Sharlin 2y ago

As mentioned elsewhere in this discussion, 99% of the time the cause is likely the use of standard C functions (or C++ `std::string`…) instead of MS's nonstandard wide versions. Which of course is a ubiquitous practice in portable command-line software like curl.

pishpash 2y ago
So the culprit is still the software writer. They should have wrapped the C++ library for OS-specific behavior on Windows. Because they are publishing buggy software and calling it cross-platform.
- bayindirh 2y ago
  curl first released in 1996, shortly after Windows 95 has born and runs on numerous Windows versions even today. So, how many different versions shall be maintained? Are you going to help one of these versions? On top of that, how many new gotchas these “modern” Windows functions hide, and how many fix cycles are required to polish them to the required level?
  - thrdbndndn 2y ago
    If we're talking about curl specifically, I absolutely think they would (NOT "should") fix/workaround it if there are actually common problems caused by it. Yes it would have required numerous fix cycles, but curl in my mind is such a polished product and they would have bit the bullet.
    - 8n4vidtmkvmk 2y ago
      Why would/should they? I've never paid for curl. Who even develops it? Sounds like a thankless job to fix obscure worstfit bugs.
      - thrdbndndn 2y ago
        Why would they develop curl at all by your logic? They fix bugs because they simply want their product to be better, if if I were to take a guess? Like, I'm sure curl's contributors worked on OS-specific problems before, and it wouldn't be the last. > to fix obscure worstfit bugs. Again my premise is "if there are actually common problems caused by it". This specific bug does sound like that, at least not for now.
      - bayindirh 2y ago
        > Why would/should they? Because they care. That's it. > I've never paid for curl. I'm sure people who develop it doesn't want money and fame, but they're just doing what they like. However, curl has commercial support contracts if you need. > Who even develops it? Daniel Stenberg et al. Daniel can be found at https://daniel.haxx.se https://daniel.haxx.se. > Sounds like a thankless job to fix obscure worstfit bugs. It may look thankless, but it's not. curl is critical infrastructure at this point. While https://xkcd.com/2347/ https://xkcd.com/2347/ applies squarely to cURL, it's actually nice that the lead developer is making some money out of his endeavor.
    - bayindirh 2y ago
      You're right, if the problems created by this are big enough, the team will fix them without any fanfare and whining. However, in neither case this is a shortcoming of curl. They'd be responding to a complicated problem caused by the platform they're running on.
- deleted 2y ago
  [deleted]
- account42 2y ago
  I'm sure those software writers will be happy to refund your purchase. It's not their fault that Microsofts standard C implementation is faulty.
smatija 2y ago
A lot of details is in linked curl hackerone: https://hackerone.com/reports/2550951 https://hackerone.com/reports/2550951
account42 2y ago
std::string is not an issue, how you get strings from the environment into it is. You can use W functions and convert the WTF-16 strings you get to WTF-8 and use that in std::string without problems.

p_ing 2y ago

> paths longer than 260 characters (if you add a "\\?\" prefix or set you manifest correctly) A long ago released build of Windows 10 did this automatically so no need for adjustments anymore, 32k is the max.... ...except for Office! It can't handle long paths. But Office has always been hacky (the title bar, for example).

comex 2y ago

_Or_ set your application to use UTF-8 for the "A" APIs. Apparently this is supported as of a Windows 10 update from 2019. [1] [1] https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...

asveikau 2y ago
It should have been supported approximately 20 years earlier than that. I was coding against Win32 looong before 2019 and wondering for years why they wouldn't let you. An explanation I heard ~10 years prior is that doing so exposed bugs in CRT and nobody wanted to fix them.
- pjmlp 2y ago
  Not sure what that has to do with CRT, given that it isn't part of Win32.
  - garganzol 2y ago
    CRT in a form of msvcrt.dll file had a de-facto presence in Windows since the end of 1990s. Later on, since 2018 or so, CRT availability was formalized in Windows API in form of ucrtbase.dll module.
    - pjmlp 2y ago
      Which doesn't change the fact that Win32 doesn't depend on it.
      - okanat 2y ago
        It is extremely hard to create an application that doesn't depend on CRT on Windows. CRT provides tables for handlers of SEH exceptions and provides the default event handlers. Win32 headers have hard dependencies of the handler tables CRT provides. So you need to go quite a bit out of your way to hack deep Win32 headers. Loading DLLs etc also may call CRT functions. You can read Mingw64 source to see how many hacks they had to do to make it work.

cesarb 2y ago

> I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000. Nowadays, it's either for historical reasons (code written back when supporting Windows 9x was important, or even code migrated from Windows 3.x), or out of a desire to support non-Windows systems. Most operating systems use a byte-based multi-byte encoding (nowadays usually UTF-8) as their native encoding, instead of UTF-16.

asveikau 2y ago

I share your recommendations of always using PWSTR when using windows apis. > I'm not sure why the non-unicode APIs are still so commonly used I think because the rest of the C world uses char* with utf-8, so that is what people are habituated to. Setting the ACP to CP_UTF8 would have solved a lot of problems, but I believe that's only been supported for a short period of time, bafflingly.

account42 2y ago
> Setting the ACP to CP_UTF8 would have solved a lot of problems, but I believe that's only been supported for a short period of time, bafflingly. It wouldn't solve all encoding problems though because most Windows APIs can store/return invalid UTF-16 which you can't represent in CP_UTF8 - you'd need a CP_WTF8 for that which doesn't even exist so you have to use the W APIs and do the conversion yourself.

ack_complete 2y ago

Using \\?\ has a downside: since it bypasses Win32's path processing, it also prevents relative paths like d:test.txt from working. Kind of annoying on the command line with tools like 7z.exe.

account42 2y ago
Sounds more like an upside TBH.

Thorrez 2y ago

>I'm not sure why the non-unicode APIs are still so commonly used. Even argv is affected on Windows. That's part of the C and C++ standard, not really a Windows API. Telling all C/C++ devs they need to stop using argv is kind of a tough ask.

shakna 2y ago

You also have to use wmain instead of main, with a wchar_t argv, otherwise the compiled-in argparser will be calling the ANSI version. In other words... Anyone using MSVC and the cross-platform standardised and normal C system, are hit by this. Oh, and wmain is a VisualC thing. It isn't found on other platforms. Not standardised.

mort96 2y ago
Writing cross platform code which consistently uses UCS-2 wchar_t* on Windows and UTF-8 char* on UNIX-like systems sounds like absolute hell
- lmz 2y ago
  A wchar_t "native" libc implementation would be an interesting thing.
- account42 2y ago
  It's not that bad really - you just convert at the win32 API call boundary. Also, it's not UCS-2. Also not UTF-16. Windows uses WTF-16 internally and if you want 100% compatibility that's what you need to target.

cryptonector 2y ago

The other opt-out might be to opt into UTF-8 support for the "A" functions.

cryptonector 2y ago

> I'm not sure why the non-unicode APIs are still so commonly used. Simple: portable code meant to run on Unix (where UTF-8 is king) and Windows -> want to use UTF-8 codepage on Windows and the "A" APIs.

account42 2y ago

> I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000. A lot of the uses are indirectly via standard C API functions that are effectively broken on Windows but work just well enough (i.e. work with ASCII) that their use goes unnoticed when someone ports something to Windows.