10 ms·

WorstFit: Unveiling Hidden Transformers in Windows ANSI

373 points by notmine1337 2y ago

mmastrac 2y ago
This is kind of unsurprising, but still new to me even as someone who did Windows development (and some Wine API hacking) for a decade around when this W/A mess came about. Windows is like the card game Munchkin, where a whole bunch of features can add up to a completely, unbelievably random over-powered exploit because of unintentional synergy between random bits. I'm happy to see that they are converting the ANSI subsystem to UTF-8, which should, in theory, mitigate a lot of these problems. I wonder if the Rust team is going to need YetAnotherFix to the process spawning API to fix this...
- ChrisSD 2y ago
  Rust's standard library basically never uses ANSI APIs. The article doesn't demonstrate any attack that works against Rust. If they do have one I'd highly recommend reporting it. Of course, Rust can't control what happens on the other side of side of a process boundary. So if an application invoked by Rust uses ANSI APIs then they'll have a problem. But also that's their responsibility.
  - okanat 2y ago
    What about the entry point? Because one of the issues mentioned in the article is about mainCRTStartup calling an ANSI API. Most of the Rust programs are linked with the C runtime. Does Rust make sure that the C library initialization is also done in Unicode APIs?
    - ChrisSD 2y ago
      No but it doesn't use C library values so it doesn't matter. E.g. getting the command line arguments is done via calling `GetCommandLineW` so it doesn't use argv or argc. This is actually necessary because Rust cannot assume it owns the entry point. E.g. a Rust library could be called from a C++ application or in a DLL, etc. So when someone calls `std::env::args` it asks the OS directly for the arguments instead of getting them from C.
mouse_ 2y ago
Unicode on modern systems is absolutely terrifying. Anyone remember the black dot of death? https://mashable.com/article/black-dot-of-death-unicode-imessage-iphone-crash https://mashable.com/article/black-dot-of-death-unicode-imes...
Randor 2y ago
That was a long read. Just be happy that you never had to deal with Trigraphs. https://learn.microsoft.com/en-us/cpp/c-language/trigraphs?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-language/trigraphs?v...
bangaladore 2y ago
I tend to agree that this is not an issue with many of the applications that are mentioned in the post. Fundamentally this boils down to essentially bugs in functions that are supposed to transform untrusted into trusted input like the example they gave: `system("wget.exe -q " . escapeshellarg($url));` `escapeshellarg` is not producing a trusted output with some certain inputs.
- blibble 2y ago
  the escaping rules for windows are so complicated (and can vary with configuration) such that it's not possible to do it securely vs. posix that just dumps the arguments directly into argv
  - hnlmorg 2y ago
    Windows doesn’t really have an ARGV though. It’s a user space abstraction for compatibility with POSIX. Windows technically just works on the principle of an executable name + a single argument. And it does this for compatibility with DOS. So you end up with this stupid escaping rules you’ve described so there are compatibility conventions at the kernel level with earlier implementations of Windows, which in turn maintained compatibility with MS-DOS. While providing a C abstraction that’s compatible with POSIX. Which is just one of many reasons why it’s a nightmare to write cross platform shells that also target Windows.
  - bangaladore 2y ago
    > the escaping rules for windows are so complicated (and can vary with configuration) such that it's not possible to do it securely This is bold claim. Is it not possible? Or not easy to do correctly?
    - blibble 2y ago
      all the kernel passes to executables is one long string and then every program handles it in whatever way it feels is best as examples: go/java/python all process arguments slightly differently even microsoft's libc changes handling between versions given it's not possible to know what parser a specific target program is going to use: it's not possible to generically serialise an array safely
vessenes 2y ago
This is a tough one. It’s systemic —- MS provides a “best fit” code mapping from wide Unicode to ASCII, which is a known, published, “vibes-based” mapper. This best fit parser is used a lottt of places, and I’m sure that it’s required for ongoing inclusion based on how MS views backward compatibility. It’s linked in by default everywhere, whether or not you know you included it. The exploits largely revolved around either speccing an unusual code point that “vibes” into say a slash or a hyphen or quotes. These code points are typically evaluated one way (correct full Unicode evaluation) inside a modern programming language, but when passed to shell commands or other Win32 API things are vibes-downed. Crucially this happens after you check them, since it’s when you’ve passed control. To quote the curl maintainer “curl is a victim” here — but who is the culprit? It seems certain that curl will be used to retrieve user supplied data automatically by a server in the future. When that server mangles user input in one way for validation and another when applied to system libraries, you’re going to have a problem. It seems to me like maybe the solution is to provide an opt-out of “best fit” munging in the Win32 space, but I’m not a Windows guy, so I speculate. At least then open source providers could just add the opt out to best practices, and deal with the many terrible problems that things like a Unicode wide variant of “ or \ delivers to them. And of course even if you do that, you’ll interact with officially shipped APIs and software that has not opted out.
- wongarsu 2y ago
  The opt-out is to use the unicode windows APIs (the functions ending in "w" instead of "a"). This also magically fixes all issues with paths longer than 260 characters (if you add a "\\?\" prefix or set you manifest correctly), and has been available and recommended since Windows XP. I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000.
  - vessenes 2y ago
    I think the issue is that native OS things like the windows command line, say, don’t always do this. Check the results of their ‘cd’ commands with Japanese Yen characters introduced. You can see that the path descriptor somehow has updated to a directory name with Yen (or a wide backslash) in it, while the file system underneath has munged, and put them into an actual directory. It’s precisely the problem that you can’t control the rest of the API surface to use W that is the source of the difficulties.
Joker_vD 2y ago
> the only thing we can do is to encourage everyone, the users, organizations, and developers, to gradually phase out ANSI and promote the use of the Wide Character API, This has been Microsoft's official position since NT 3.5, if I remember correctly. Sadly, one of the main hurdles is the way Microsoft's own C/C++ runtime library (msvcrt.dll) is implemented. Its non-standard "wide" functions like _wfopen(), _wgetenv(), etc. internally use W-functions from Win API. But the standard, "narrow" functions like fopen(), getenv(), etc., instead of using the "wide" versions and converting to-from Unicode themselves (and reporting conversion failures), simply use A-functions. Which, as you see, generally don't report any Unicode conversion failures but instead try to gloss over them using best-fit approach. And of course, nobody who ports software, written in C, to Windows wants to rewrite all of the uses of standard functions to use Microsoft's non-portable functions because at this point, it becomes a full-blown rewrite.
- delta_p_delta_x 2y ago
  > Microsoft's own C/C++ runtime library (msvcrt.dll) is implemented This has been superseded by the Universal C runtime (UCRT)[1] which is C99-compliant.
  - pjmlp 2y ago
    Mostly C99 compliant, some things are left out. https://learn.microsoft.com/en-us/cpp/c-runtime-library/compatibility?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-runtime-library/comp...
    - sigsev_251 2y ago
      I think the documentation is outdated given that C11 atomics [1] and threads [2] are available for more than a year now. Same goes for pretty much everything MSVC frontend related stuff (I've yet to try which C++23 features are supported at the moment, but they've secretly added support for C23 features like typeof and attributes, as well as GNU Statement Expressions). [1]: https://devblogs.microsoft.com/cppblog/c11-atomics-in-visual-studio-2022-version-17-5-preview-2/ https://devblogs.microsoft.com/cppblog/c11-atomics-in-visual... [2]: https://devblogs.microsoft.com/cppblog/c11-threads-in-visual-studio-2022-version-17-8-preview-2/ https://devblogs.microsoft.com/cppblog/c11-threads-in-visual...
segasaturn 2y ago
I've been inadvertantly safe from this bug on my personal Windows computer for years thanks to having the UTF-8 mode set, as shown at the bottom of the article. I had it set due to some old, foreign games showing garbled nonsense text on my computer. Have not noticed any bugs or side effects despite it being labelled as "Beta".
- numpad0 2y ago
  Interesting, to me that checkbox have done nothing but crashing too many random apps. I guess whether it works depends on the user's home codepage with it off.
- UltraSane 2y ago
  I just enabled the "Beta: Use Unicode UTF-8 for worldwide language support" option. Going to be interesting to see how many apps this breaks.
  - cryptonector 2y ago
    Please come back and tell us!
    - UltraSane 2y ago
      Nothing so far.
ok123456 2y ago
Bush hid the facts
- deleted 2y ago
  [deleted]
- cesarb 2y ago
  > Bush hid the facts For those who don't know the reference: https://en.wikipedia.org/wiki/Bush_hid_the_facts https://en.wikipedia.org/wiki/Bush_hid_the_facts it's a vaguely related issue, in which a Windows component misinterprets a sequence of ASCII characters as a sequence of UTF-16 characters. Windows just seems full of these sorts of character-handling bugs, in part due to its long history as a descendant of the codepage-using MS-DOS and 16-bit Windows operating systems.
cesarb 2y ago
> However, resolving this problem isn’t that as simple as just replacing the main() with its wide-character counterpart. Since the function signature has been changed, maintainers would need to rewrite all variable definitions and argument parsing logics, converting everything from simple char * to wchar_t *. This process can be painful and error-prone. You don't need to convert everything from char * to wchar *. You can instead convert the wide characters you received to UTF-8 (or to something like Rust's WTF-8, if you want to also allow invalid sequences like unpaired surrogates), and keep using "char" everywhere; of course, you have to take care to not mix ANSI or OEMCP strings with UTF-8 strings, which is easy if you simply use UTF-8 everywhere. This is the approach advocated by the classic https://utf8everywhere.org/ https://utf8everywhere.org/ site.
Dwedit 2y ago
There are two ways to force the "Ansi" codepage to actually be UTF-8 for an application that you write (or an EXE that you patch). One way is with a Manifest file, and works as of a particular build of Windows 10. This can also be applied to any EXE after building it. So if you want a program to gain UTF-8 support, you can hack it in. Most useful for console-mode programs. The other way is to use the hacks that "App Locale" type tools use. One way involves undocumented function calls from NTDLL. I'm not sure exactly which functions you need to call, but I think it might involve "RtlInitNlsTables" and "RtlResetRtlTranslations" (not actually sure).
layer8 2y ago
> And yes, Python’s subprocess module can’t prevent this. A reasonably sane solution would be for it to reject command line arguments on Windows that contain non-ASCII characters or ASCII characters that aren’t portable across code pages (not all code pages are a superset of US-ASCII), by default, and to support an optional parameter to allow the full range, documenting the risk.
- veltas 2y ago
  That's not sane at all, because then you can't send e.g. Japanese arguments to tools that support wide chars or have UTF-8 codepage in their manifest. And then there's yet another difference between Python versions to trip you up. And why should the default not allow internationalisation? Doesn't fit with the idea of Python 3.
layer8 2y ago
> until Microsoft chooses to enable UTF-8 by default in all of their Windows editions. I don’t know how likely this is. There are a lot of old applications that assume a particular code page, or assume 1 byte per character, that this would break. There are also more subtle variations of this, like applications assuming that converting from wide characters to ANSI can’t increase the number of bytes (and hence an existing buffer can be safely reused), which isn’t the case for UTF-8 (but for all, or almost all, existing code pages). It can open up new vulnerabilities. It would probably cause much less breakage to remove the Best-Fit logic from the win32 xxxA APIs, and instead have all unmappable characters be replaced by a character without any common meta semantics, like “x”.
- kgeist 2y ago
  Maybe they can introduce OS API versions (if there's no such thing yet) and require new (or updated) apps targetting new API versions/newer SDKs to assume UTF8 by default? So everything below a certain API version is emulated legacy mode. Windows already has the concept of shims to emulate behavior of different Windows versions.
  - layer8 2y ago
    Apps can already opt-in to UTF-8 for the ANSI APIs (see https://news.ycombinator.com/item?id=42649122 https://news.ycombinator.com/item?id=42649122), or use the wide-character APIs.
- tambre 2y ago
  One example of such an application is Adobe After Effects [0]. Or at least used to be, I no longer use Windows. [0] https://tambre.ee/blog/adobe_after_effects_windows_utf-8/ https://tambre.ee/blog/adobe_after_effects_windows_utf-8/
- cryptonector 2y ago
  You already had this problem pre-UTF-8 in Windows: changing your default codepage could cause app fuckiness. So giving the user the option to use UTF-8 is reasonable. Making it the default is also reasonable given the problems that the Best-Fit mapping are causing, though Microsoft would have to do something to help users easily figure out how to run older code. Another not-so-reasonable thing would be to drop all mappings to "special" ASCII characters from the Best-Fit mappings, though this wouldn't help apps linked statically with the CRT. Not so reasonable because it doesn't fix the vulnerabilities. Sometimes security vulnerabilities motivate backwards-compatibility breakage.
scoopr 2y ago
I was wondering if the beta checkbox the same thing as setting the ActiveCodePage to UTF-8 in the manifest, but the docs[0] clarify that GDI doesn't adhere to per-process codepage, but only a single global one, which is what the checkbox sets. Bit of a shame that you can't fully opt-in to be UTF-8 with the *A API, for your own apps. But I think for the issues highlighted in the post, I think it would still be a valid workaround/defence-in-depth thing. [0] https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...
ppp999 2y ago
Character encoding has been such a mess for so long it's crazy.
- Aachen 2y ago
  To be fair, I do enjoy that you can start programs like Rollercoaster Tycoon, written mostly in assembly, still in Windows 7 (maybe even newer, I haven't used Windows in a while) and it mostly all works. I'm not aware that there are characters that can't be represented by the UTF-16 encoding (that will, according to the article, also widen further when necessary) which has been in use for decades now. I don't feel like this is character encoding still being a mess as much as legacy causing a mess: without those legacy binaries, translation wouldn't be needed and there wouldn't be a problem since the newer encoding is long there I find encoding to be a mostly solved problem in any software that supports /UTF-[0-9]+/. Can't remember the last time I ran into an encoding issue, maybe it was when I ran PHP on a Windows server years and years ago, which defaulted to passing query parameters in ISO-8859-15 or something, which didn't work with html escaping (returned a blank string, so failing in a safe and obvious way iirc) if you didn't specify the character set. I personally converted, or set, everything I create or use to UTF-8 by now
lilyball 2y ago
> Worse still, as the attack exploits behavior at the system level during the conversion process, no standard library in any programming language can fully stop our attack! What happens if the standard library updates its shell escaping to also escape things like the Yen character and any other character that has a Best-Fit translation into a quote or backslash? Which is to say, what does Windows do for command-line splitting if it encounters a backslash-escaped nonspecial character in a quoted string? If it behaves like sh and the backslash simply disables special handling of the next character, then backslash-escaping any threat characters should work.
- david2ndaccount 2y ago
  Backslash is a valid path character (you can use / or \ as the path separator on windows) so if the backslash isn’t actually escaping anything it is left as-is.
- cryptonector 2y ago
  > What happens if the standard library updates its shell escaping If the executable is linked statically with the CRT then nothing changes until you re-link it with the newer CRT. If it links with the UCRT then if the UCRT changes its rules then the program will too.
tiahura 2y ago
Imagine no Unicode, It’s easy if you try, No bytes that bloat our systems, No errors make us cry. Imagine all the coders, Living life in ASCII… Imagine no emojis, Just letters, plain and true, No accents to confuse us, No glyphs in Sanskrit too. Imagine all the programs, Running clean and fast… You may say I’m a dreamer, But I’m not the only one. I hope someday you’ll join us, And encoding wars will be done.
- Ndymium 2y ago
  I can appreciate the funny lyrics, but in real life I appreciate being able to write in my own language on a computer. Or even a mix of my language and another non-English language!
  - lmm 2y ago
    > in real life I appreciate being able to write in my own language on a computer. I do too, which is why I hate the "unicode only for everything everywhere" narrative that's taken hold. My language can't be written properly in Unicode, so support for traditional codepages and encodings is really important!
    - klibertp 2y ago
      What language is that? Given that there are glyphs for Ancient Egyptian in Unicode now, the claim that there's a language you can't write in Unicode needs some backing :)
      - Ndymium 2y ago
        I presume they mean Han unification[0], which to an uneducated outsider sure sounds like a fuckup. I wonder if it's possible to fix it inside Unicode without throwing the rest of it away. Reserve new blocks for all the needed graphemes and deprecate the unified ones? [0] https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
      - lmm 2y ago
nitwit005 2y ago
There are presumably some similar .Net COM issues when communicating with unmanaged code, as there is an attribute for controlling this conversion: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.bestfitmappingattribute https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... It directly mentions: "Setting BestFitMappingAttribute parameters in this manner provides an added measure of security."
rubatuga 2y ago
From what I can tell the largest vulnerability is argument passing to executables in Windows. Essentially it is very difficult to safeguard it. I've seen some CLI programs use the '--' to signify user input at the end, maybe this would solve this for a single argument scenario. Overall, this is an excellent article and vulnerability discovery.
EdSharkey 2y ago
Distributing native binaries is so dangerous!
- garganzol 2y ago
  Any executable code is dangerous if isolation assumptions are violated. JavaScript, Python, or anything Turing-complete. It does not matter if the code is native or interpreted.
  - perching_aix 2y ago
    Distributing any data is dangerous. Programs and devices have been compromised by malicious text, audio, images, video and other kinds of binary media (data), so even (especially?) programs. So I'd argue this - on it's own - is not really a useful thing to remark. The question is the characteristics and interactions of the dangers. The comment you responded to is weird because of exactly this reason. Very terse without a whole lot of substance, coming off as interaction bait / similar. All too often do I see comments on social media where people post something that will obviously lead others into being cornered arguing something that was misleading in the first place. Best not to entertain these unless confidence can be had that they're being genuine and are phrasing like this by mistake / not knowing better.
UltraSane 2y ago
The loosey-goosey mapping of code points to characters has always bothered me about Unicode. To guard against this nasty issue that is going to take years to fix you can enable global UTF-8 support by doing Settings > Time & language > Language & region > Administrative language settings > Change system locale, and check Beta: Use Unicode UTF-8 for worldwide language support. Then reboot the PC for the change to take effect.
- cryptonector 2y ago
  What "loosey-goosey mapping of code points to characters" are you talking about?
  - perching_aix 2y ago
    I'd imagine the same one the article is talking about [0]. [0] https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/readme.txt https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/Windo...
    - cryptonector 2y ago
      Thanks for that link (bookmarked). GP Had written: | The loosey-goosey mapping of code points to characters has always bothered me about Unicode. but the ones you posted are Microsoft's Best-Fit mappings. I'm going to conclude that GP referred to just that.
est 2y ago
I remember typing some prefix character in notepad.exe then your hole txt became messed up. Funny unicode times.
- jeroenhd 2y ago
  That didn't have anything to do with the mechanism at play here. The "bush hid the facts" example interfered with the unicode detection mechanism (IsTextUnicode) built into Notepad itself. If it just defaulted to the current code page, it wouldn't have had this problem, but because IsTextUnicode misdetects various word length sequences, it needlessly converted the file from ASCII into UTF-16LE, which would often produce Chinese characters by chance. https://devblogs.microsoft.com/oldnewthing/20040324-00/?p=40093 https://devblogs.microsoft.com/oldnewthing/20040324-00/?p=40...
LudwigNagasena 2y ago
What would even be the proper way to do `system("wget.exe -q " . escapeshellarg($url))`? It’s ridiculous that plaintext IPC is still the primary interface for many tools.
- rubatuga 2y ago
  Agreed, Windows should just make some breaking changes already and adopt unix style arguments.
  - ygra 2y ago
    So we end up in a world 20 years from now where most applications still don't use that. I guess the main problem as described here is the mapping, as argument splitting was just one of the possible things that break (next to argument validation or bad file names).
- cryptonector 2y ago
  Parse the URI query parameters and construct the command-line. "Parse, don't validate." Though still, that's just not enough here. If the command is a "main()" style command then you'll lose. You'll need to make sure that the command is safe to use with Unicode data regardless of codepage in use.
lifthrasiir 2y ago
Oh, my, freaking, god. I knew Windows API provides that sort of best-fit conversions, but didn't realize that it was a default behavior for several ANSI functions in my native code page (949 [1])! At this point they should be just banned like gets. [1] Yes, I know there is a UTF-8 code page (65001). That was really unusable for a long time and still is suffering compatibility issues to this day.
kazinator 2y ago
HN, Help! Before I dive into this, does anyone know whether this affects the argument parsing in Cygwin, that prepares the arguments for a regular int main(int argc, char *argv)? TXR Lisp uses wchar_t strings, and the "W" functions on Windows. So that's well and good. But it does start with a regular C main, relying on the Cygwin run-time for that. If that's vulnerable, I will hack it to have its own argument parsing, using the wide char command line. Maybe I should ask this on the Cygwin mailing list.
- shakna 2y ago
  Cygwin has been using the W variant of most things. [0] Unfortunately, they're using it on "__argv", not "__wargv". Which means they are probably vulnerable to this. [0] https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/kernel32.cc#n417 https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/ker...
- cryptonector 2y ago
  Starting with main() instead of wmain() is enough to make it vulnerable :(
  - kazinator 2y ago
    It's enough to make it suspicious, but there is a way for a C compiler on Windows to support regular main startup such that it is not vulnerable. It depends on which/whose piece of code takes the WCHAR-based command line and produces the char-based argv[]. Does that code do the "WorstFit" thing, or not. In my program, I could fix it (if necessary) by taking the WCHAR command line and parsing it myself, using my own UTF8 encoding routines to make the multi-byte strings.
    - cryptonector 2y ago
      Yes, that's what I'd do: write a `wmain()` that converts UTF-16 arguments to UTF-8 then calls the real `main()` (which you'll have to rename).
JimmyWilliams1 2y ago
[dead]
garganzol 2y ago
Microsoft was aware of this issue at least 1 year ago. I know this because they released a special code analysis rule CA2101 [1] that explicitly discouraged the use of the best-fit mapping. They mentioned security vulnerabilities in the rule’s description, but they were purposefully vague in details though. [1] https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2101 https://learn.microsoft.com/en-us/dotnet/fundamentals/code-a...
sharpshadow 2y ago
That’s amazing great read. According to for example this[0] post it’s possible to change code pages in windows in various ways and would allow the use of multiple BestFit scenarios on the same OS without reboot. Even combining them should be possible.
- Aachen 2y ago
  (you forgot the link)
  - sharpshadow 2y ago
    Right. https://stackoverflow.com/a/56323296 https://stackoverflow.com/a/56323296
pornel 2y ago
It would be easily fixable if CommandlineToArgvA was obtaining the command line itself. Then instead of converting to ANSI and then parsing that, it could parse args in Unicode, and then convert argument by argument to ANSI. The output would be ANSI compatible, but split and unescaped in the true form. Unfortunately, the parsing is a two-step operation, with the application calling GetCommandLineA itself first and passing that to the parser, so a fix would need a hack to correlate the versions of the command line input without breaking when it's given a different string.
radarsat1 2y ago
Seems like a another possible fix would be to change the best fit mapping table to never generate control characters, but only alphanumerics. So map quote-like characters to 'q' and so on. This might be uglier and slightly change behaviour, but only for vulnerable applications.
account42 2y ago
Window's A APIs and conversion functions are best ingored entirely. Always use W functions and use your own converions (that can round-trip invalid UTF-16 like WTF-8) if you want to use an 8-bit encoding internally. Most (all?) of the exploits here are already bugs because the applications don't properly handle unicode data.

373 points by notmine1337 2y ago

This is kind of unsurprising, but still new to me even as someone who did Windows development (and some Wine API hacking) for a decade around when this W/A mess came about. Windows is like the card game Munchkin, where a whole bunch of features can add up to a completely, unbelievably random over-powered exploit because of unintentional synergy between random bits. I'm happy to see that they are converting the ANSI subsystem to UTF-8, which should, in theory, mitigate a lot of these problems. I wonder if the Rust team is going to need YetAnotherFix to the process spawning API to fix this...

ChrisSD 2y ago
Rust's standard library basically never uses ANSI APIs. The article doesn't demonstrate any attack that works against Rust. If they do have one I'd highly recommend reporting it. Of course, Rust can't control what happens on the other side of side of a process boundary. So if an application invoked by Rust uses ANSI APIs then they'll have a problem. But also that's their responsibility.
- okanat 2y ago
  What about the entry point? Because one of the issues mentioned in the article is about mainCRTStartup calling an ANSI API. Most of the Rust programs are linked with the C runtime. Does Rust make sure that the C library initialization is also done in Unicode APIs?
  - ChrisSD 2y ago
    No but it doesn't use C library values so it doesn't matter. E.g. getting the command line arguments is done via calling `GetCommandLineW` so it doesn't use argv or argc. This is actually necessary because Rust cannot assume it owns the entry point. E.g. a Rust library could be called from a C++ application or in a DLL, etc. So when someone calls `std::env::args` it asks the OS directly for the arguments instead of getting them from C.

mouse_ 2y ago

Unicode on modern systems is absolutely terrifying. Anyone remember the black dot of death? https://mashable.com/article/black-dot-of-death-unicode-imessage-iphone-crash https://mashable.com/article/black-dot-of-death-unicode-imes...

Randor 2y ago

That was a long read. Just be happy that you never had to deal with Trigraphs. https://learn.microsoft.com/en-us/cpp/c-language/trigraphs?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-language/trigraphs?v...

bangaladore 2y ago

I tend to agree that this is not an issue with many of the applications that are mentioned in the post. Fundamentally this boils down to essentially bugs in functions that are supposed to transform untrusted into trusted input like the example they gave: `system("wget.exe -q " . escapeshellarg($url));` `escapeshellarg` is not producing a trusted output with some certain inputs.

blibble 2y ago
the escaping rules for windows are so complicated (and can vary with configuration) such that it's not possible to do it securely vs. posix that just dumps the arguments directly into argv
- hnlmorg 2y ago
  Windows doesn’t really have an ARGV though. It’s a user space abstraction for compatibility with POSIX. Windows technically just works on the principle of an executable name + a single argument. And it does this for compatibility with DOS. So you end up with this stupid escaping rules you’ve described so there are compatibility conventions at the kernel level with earlier implementations of Windows, which in turn maintained compatibility with MS-DOS. While providing a C abstraction that’s compatible with POSIX. Which is just one of many reasons why it’s a nightmare to write cross platform shells that also target Windows.
- bangaladore 2y ago
  > the escaping rules for windows are so complicated (and can vary with configuration) such that it's not possible to do it securely This is bold claim. Is it not possible? Or not easy to do correctly?
  - blibble 2y ago
    all the kernel passes to executables is one long string and then every program handles it in whatever way it feels is best as examples: go/java/python all process arguments slightly differently even microsoft's libc changes handling between versions given it's not possible to know what parser a specific target program is going to use: it's not possible to generically serialise an array safely

vessenes 2y ago

This is a tough one. It’s systemic —- MS provides a “best fit” code mapping from wide Unicode to ASCII, which is a known, published, “vibes-based” mapper. This best fit parser is used a lottt of places, and I’m sure that it’s required for ongoing inclusion based on how MS views backward compatibility. It’s linked in by default everywhere, whether or not you know you included it. The exploits largely revolved around either speccing an unusual code point that “vibes” into say a slash or a hyphen or quotes. These code points are typically evaluated one way (correct full Unicode evaluation) inside a modern programming language, but when passed to shell commands or other Win32 API things are vibes-downed. Crucially this happens after you check them, since it’s when you’ve passed control. To quote the curl maintainer “curl is a victim” here — but who is the culprit? It seems certain that curl will be used to retrieve user supplied data automatically by a server in the future. When that server mangles user input in one way for validation and another when applied to system libraries, you’re going to have a problem. It seems to me like maybe the solution is to provide an opt-out of “best fit” munging in the Win32 space, but I’m not a Windows guy, so I speculate. At least then open source providers could just add the opt out to best practices, and deal with the many terrible problems that things like a Unicode wide variant of “ or \ delivers to them. And of course even if you do that, you’ll interact with officially shipped APIs and software that has not opted out.

wongarsu 2y ago
The opt-out is to use the unicode windows APIs (the functions ending in "w" instead of "a"). This also magically fixes all issues with paths longer than 260 characters (if you add a "\\?\" prefix or set you manifest correctly), and has been available and recommended since Windows XP. I'm not sure why the non-unicode APIs are still so commonly used. I can't imagine it's out of a desire to support Windows 98 or Window 2000.
- vessenes 2y ago
  I think the issue is that native OS things like the windows command line, say, don’t always do this. Check the results of their ‘cd’ commands with Japanese Yen characters introduced. You can see that the path descriptor somehow has updated to a directory name with Yen (or a wide backslash) in it, while the file system underneath has munged, and put them into an actual directory. It’s precisely the problem that you can’t control the rest of the API surface to use W that is the source of the difficulties.

Joker_vD 2y ago

> the only thing we can do is to encourage everyone, the users, organizations, and developers, to gradually phase out ANSI and promote the use of the Wide Character API, This has been Microsoft's official position since NT 3.5, if I remember correctly. Sadly, one of the main hurdles is the way Microsoft's own C/C++ runtime library (msvcrt.dll) is implemented. Its non-standard "wide" functions like _wfopen(), _wgetenv(), etc. internally use W-functions from Win API. But the standard, "narrow" functions like fopen(), getenv(), etc., instead of using the "wide" versions and converting to-from Unicode themselves (and reporting conversion failures), simply use A-functions. Which, as you see, generally don't report any Unicode conversion failures but instead try to gloss over them using best-fit approach. And of course, nobody who ports software, written in C, to Windows wants to rewrite all of the uses of standard functions to use Microsoft's non-portable functions because at this point, it becomes a full-blown rewrite.

delta_p_delta_x 2y ago
> Microsoft's own C/C++ runtime library (msvcrt.dll) is implemented This has been superseded by the Universal C runtime (UCRT)[1] which is C99-compliant.
- pjmlp 2y ago
  Mostly C99 compliant, some things are left out. https://learn.microsoft.com/en-us/cpp/c-runtime-library/compatibility?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-runtime-library/comp...
  - sigsev_251 2y ago
    I think the documentation is outdated given that C11 atomics [1] and threads [2] are available for more than a year now. Same goes for pretty much everything MSVC frontend related stuff (I've yet to try which C++23 features are supported at the moment, but they've secretly added support for C23 features like typeof and attributes, as well as GNU Statement Expressions). [1]: https://devblogs.microsoft.com/cppblog/c11-atomics-in-visual-studio-2022-version-17-5-preview-2/ https://devblogs.microsoft.com/cppblog/c11-atomics-in-visual... [2]: https://devblogs.microsoft.com/cppblog/c11-threads-in-visual-studio-2022-version-17-8-preview-2/ https://devblogs.microsoft.com/cppblog/c11-threads-in-visual...

segasaturn 2y ago

I've been inadvertantly safe from this bug on my personal Windows computer for years thanks to having the UTF-8 mode set, as shown at the bottom of the article. I had it set due to some old, foreign games showing garbled nonsense text on my computer. Have not noticed any bugs or side effects despite it being labelled as "Beta".

numpad0 2y ago
Interesting, to me that checkbox have done nothing but crashing too many random apps. I guess whether it works depends on the user's home codepage with it off.
UltraSane 2y ago
I just enabled the "Beta: Use Unicode UTF-8 for worldwide language support" option. Going to be interesting to see how many apps this breaks.
- cryptonector 2y ago
  Please come back and tell us!
  - UltraSane 2y ago
    Nothing so far.

ok123456 2y ago

Bush hid the facts

deleted 2y ago
[deleted]
cesarb 2y ago
> Bush hid the facts For those who don't know the reference: https://en.wikipedia.org/wiki/Bush_hid_the_facts https://en.wikipedia.org/wiki/Bush_hid_the_facts it's a vaguely related issue, in which a Windows component misinterprets a sequence of ASCII characters as a sequence of UTF-16 characters. Windows just seems full of these sorts of character-handling bugs, in part due to its long history as a descendant of the codepage-using MS-DOS and 16-bit Windows operating systems.

cesarb 2y ago

> However, resolving this problem isn’t that as simple as just replacing the main() with its wide-character counterpart. Since the function signature has been changed, maintainers would need to rewrite all variable definitions and argument parsing logics, converting everything from simple char * to wchar_t *. This process can be painful and error-prone. You don't need to convert everything from char * to wchar *. You can instead convert the wide characters you received to UTF-8 (or to something like Rust's WTF-8, if you want to also allow invalid sequences like unpaired surrogates), and keep using "char" everywhere; of course, you have to take care to not mix ANSI or OEMCP strings with UTF-8 strings, which is easy if you simply use UTF-8 everywhere. This is the approach advocated by the classic https://utf8everywhere.org/ https://utf8everywhere.org/ site.

Dwedit 2y ago

There are two ways to force the "Ansi" codepage to actually be UTF-8 for an application that you write (or an EXE that you patch). One way is with a Manifest file, and works as of a particular build of Windows 10. This can also be applied to any EXE after building it. So if you want a program to gain UTF-8 support, you can hack it in. Most useful for console-mode programs. The other way is to use the hacks that "App Locale" type tools use. One way involves undocumented function calls from NTDLL. I'm not sure exactly which functions you need to call, but I think it might involve "RtlInitNlsTables" and "RtlResetRtlTranslations" (not actually sure).

layer8 2y ago

> And yes, Python’s subprocess module can’t prevent this. A reasonably sane solution would be for it to reject command line arguments on Windows that contain non-ASCII characters or ASCII characters that aren’t portable across code pages (not all code pages are a superset of US-ASCII), by default, and to support an optional parameter to allow the full range, documenting the risk.

veltas 2y ago
That's not sane at all, because then you can't send e.g. Japanese arguments to tools that support wide chars or have UTF-8 codepage in their manifest. And then there's yet another difference between Python versions to trip you up. And why should the default not allow internationalisation? Doesn't fit with the idea of Python 3.

layer8 2y ago

> until Microsoft chooses to enable UTF-8 by default in all of their Windows editions. I don’t know how likely this is. There are a lot of old applications that assume a particular code page, or assume 1 byte per character, that this would break. There are also more subtle variations of this, like applications assuming that converting from wide characters to ANSI can’t increase the number of bytes (and hence an existing buffer can be safely reused), which isn’t the case for UTF-8 (but for all, or almost all, existing code pages). It can open up new vulnerabilities. It would probably cause much less breakage to remove the Best-Fit logic from the win32 xxxA APIs, and instead have all unmappable characters be replaced by a character without any common meta semantics, like “x”.

kgeist 2y ago
Maybe they can introduce OS API versions (if there's no such thing yet) and require new (or updated) apps targetting new API versions/newer SDKs to assume UTF8 by default? So everything below a certain API version is emulated legacy mode. Windows already has the concept of shims to emulate behavior of different Windows versions.
- layer8 2y ago
  Apps can already opt-in to UTF-8 for the ANSI APIs (see https://news.ycombinator.com/item?id=42649122 https://news.ycombinator.com/item?id=42649122), or use the wide-character APIs.
tambre 2y ago
One example of such an application is Adobe After Effects [0]. Or at least used to be, I no longer use Windows. [0] https://tambre.ee/blog/adobe_after_effects_windows_utf-8/ https://tambre.ee/blog/adobe_after_effects_windows_utf-8/
cryptonector 2y ago
You already had this problem pre-UTF-8 in Windows: changing your default codepage could cause app fuckiness. So giving the user the option to use UTF-8 is reasonable. Making it the default is also reasonable given the problems that the Best-Fit mapping are causing, though Microsoft would have to do something to help users easily figure out how to run older code. Another not-so-reasonable thing would be to drop all mappings to "special" ASCII characters from the Best-Fit mappings, though this wouldn't help apps linked statically with the CRT. Not so reasonable because it doesn't fix the vulnerabilities. Sometimes security vulnerabilities motivate backwards-compatibility breakage.

scoopr 2y ago

I was wondering if the beta checkbox the same thing as setting the ActiveCodePage to UTF-8 in the manifest, but the docs[0] clarify that GDI doesn't adhere to per-process codepage, but only a single global one, which is what the checkbox sets. Bit of a shame that you can't fully opt-in to be UTF-8 with the *A API, for your own apps. But I think for the issues highlighted in the post, I think it would still be a valid workaround/defence-in-depth thing. [0] https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...

ppp999 2y ago

Character encoding has been such a mess for so long it's crazy.

Aachen 2y ago
To be fair, I do enjoy that you can start programs like Rollercoaster Tycoon, written mostly in assembly, still in Windows 7 (maybe even newer, I haven't used Windows in a while) and it mostly all works. I'm not aware that there are characters that can't be represented by the UTF-16 encoding (that will, according to the article, also widen further when necessary) which has been in use for decades now. I don't feel like this is character encoding still being a mess as much as legacy causing a mess: without those legacy binaries, translation wouldn't be needed and there wouldn't be a problem since the newer encoding is long there I find encoding to be a mostly solved problem in any software that supports /UTF-[0-9]+/. Can't remember the last time I ran into an encoding issue, maybe it was when I ran PHP on a Windows server years and years ago, which defaulted to passing query parameters in ISO-8859-15 or something, which didn't work with html escaping (returned a blank string, so failing in a safe and obvious way iirc) if you didn't specify the character set. I personally converted, or set, everything I create or use to UTF-8 by now

lilyball 2y ago

> Worse still, as the attack exploits behavior at the system level during the conversion process, no standard library in any programming language can fully stop our attack! What happens if the standard library updates its shell escaping to also escape things like the Yen character and any other character that has a Best-Fit translation into a quote or backslash? Which is to say, what does Windows do for command-line splitting if it encounters a backslash-escaped nonspecial character in a quoted string? If it behaves like sh and the backslash simply disables special handling of the next character, then backslash-escaping any threat characters should work.

david2ndaccount 2y ago
Backslash is a valid path character (you can use / or \ as the path separator on windows) so if the backslash isn’t actually escaping anything it is left as-is.
cryptonector 2y ago
> What happens if the standard library updates its shell escaping If the executable is linked statically with the CRT then nothing changes until you re-link it with the newer CRT. If it links with the UCRT then if the UCRT changes its rules then the program will too.

tiahura 2y ago

Imagine no Unicode, It’s easy if you try, No bytes that bloat our systems, No errors make us cry. Imagine all the coders, Living life in ASCII… Imagine no emojis, Just letters, plain and true, No accents to confuse us, No glyphs in Sanskrit too. Imagine all the programs, Running clean and fast… You may say I’m a dreamer, But I’m not the only one. I hope someday you’ll join us, And encoding wars will be done.

Ndymium 2y ago
I can appreciate the funny lyrics, but in real life I appreciate being able to write in my own language on a computer. Or even a mix of my language and another non-English language!
- lmm 2y ago
  > in real life I appreciate being able to write in my own language on a computer. I do too, which is why I hate the "unicode only for everything everywhere" narrative that's taken hold. My language can't be written properly in Unicode, so support for traditional codepages and encodings is really important!
  - klibertp 2y ago
    What language is that? Given that there are glyphs for Ancient Egyptian in Unicode now, the claim that there's a language you can't write in Unicode needs some backing :)
    - Ndymium 2y ago
      I presume they mean Han unification[0], which to an uneducated outsider sure sounds like a fuckup. I wonder if it's possible to fix it inside Unicode without throwing the rest of it away. Reserve new blocks for all the needed graphemes and deprecate the unified ones? [0] https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
    - lmm 2y ago

nitwit005 2y ago

There are presumably some similar .Net COM issues when communicating with unmanaged code, as there is an attribute for controlling this conversion: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.bestfitmappingattribute https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... It directly mentions: "Setting BestFitMappingAttribute parameters in this manner provides an added measure of security."

rubatuga 2y ago

From what I can tell the largest vulnerability is argument passing to executables in Windows. Essentially it is very difficult to safeguard it. I've seen some CLI programs use the '--' to signify user input at the end, maybe this would solve this for a single argument scenario. Overall, this is an excellent article and vulnerability discovery.

EdSharkey 2y ago

Distributing native binaries is so dangerous!

garganzol 2y ago
Any executable code is dangerous if isolation assumptions are violated. JavaScript, Python, or anything Turing-complete. It does not matter if the code is native or interpreted.
- perching_aix 2y ago
  Distributing any data is dangerous. Programs and devices have been compromised by malicious text, audio, images, video and other kinds of binary media (data), so even (especially?) programs. So I'd argue this - on it's own - is not really a useful thing to remark. The question is the characteristics and interactions of the dangers. The comment you responded to is weird because of exactly this reason. Very terse without a whole lot of substance, coming off as interaction bait / similar. All too often do I see comments on social media where people post something that will obviously lead others into being cornered arguing something that was misleading in the first place. Best not to entertain these unless confidence can be had that they're being genuine and are phrasing like this by mistake / not knowing better.

UltraSane 2y ago

The loosey-goosey mapping of code points to characters has always bothered me about Unicode. To guard against this nasty issue that is going to take years to fix you can enable global UTF-8 support by doing Settings > Time & language > Language & region > Administrative language settings > Change system locale, and check Beta: Use Unicode UTF-8 for worldwide language support. Then reboot the PC for the change to take effect.

cryptonector 2y ago
What "loosey-goosey mapping of code points to characters" are you talking about?
- perching_aix 2y ago
  I'd imagine the same one the article is talking about [0]. [0] https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/readme.txt https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/Windo...
  - cryptonector 2y ago
    Thanks for that link (bookmarked). GP Had written: | The loosey-goosey mapping of code points to characters has always bothered me about Unicode. but the ones you posted are Microsoft's Best-Fit mappings. I'm going to conclude that GP referred to just that.

est 2y ago

I remember typing some prefix character in notepad.exe then your hole txt became messed up. Funny unicode times.

jeroenhd 2y ago
That didn't have anything to do with the mechanism at play here. The "bush hid the facts" example interfered with the unicode detection mechanism (IsTextUnicode) built into Notepad itself. If it just defaulted to the current code page, it wouldn't have had this problem, but because IsTextUnicode misdetects various word length sequences, it needlessly converted the file from ASCII into UTF-16LE, which would often produce Chinese characters by chance. https://devblogs.microsoft.com/oldnewthing/20040324-00/?p=40093 https://devblogs.microsoft.com/oldnewthing/20040324-00/?p=40...

LudwigNagasena 2y ago

What would even be the proper way to do `system("wget.exe -q " . escapeshellarg($url))`? It’s ridiculous that plaintext IPC is still the primary interface for many tools.

rubatuga 2y ago
Agreed, Windows should just make some breaking changes already and adopt unix style arguments.
- ygra 2y ago
  So we end up in a world 20 years from now where most applications still don't use that. I guess the main problem as described here is the mapping, as argument splitting was just one of the possible things that break (next to argument validation or bad file names).
cryptonector 2y ago
Parse the URI query parameters and construct the command-line. "Parse, don't validate." Though still, that's just not enough here. If the command is a "main()" style command then you'll lose. You'll need to make sure that the command is safe to use with Unicode data regardless of codepage in use.

lifthrasiir 2y ago

Oh, my, freaking, god. I knew Windows API provides that sort of best-fit conversions, but didn't realize that it was a default behavior for several ANSI functions in my native code page (949 [1])! At this point they should be just banned like gets. [1] Yes, I know there is a UTF-8 code page (65001). That was really unusable for a long time and still is suffering compatibility issues to this day.

kazinator 2y ago

HN, Help! Before I dive into this, does anyone know whether this affects the argument parsing in Cygwin, that prepares the arguments for a regular int main(int argc, char *argv)? TXR Lisp uses wchar_t strings, and the "W" functions on Windows. So that's well and good. But it does start with a regular C main, relying on the Cygwin run-time for that. If that's vulnerable, I will hack it to have its own argument parsing, using the wide char command line. Maybe I should ask this on the Cygwin mailing list.

shakna 2y ago
Cygwin has been using the W variant of most things. [0] Unfortunately, they're using it on "__argv", not "__wargv". Which means they are probably vulnerable to this. [0] https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/kernel32.cc#n417 https://cygwin.com/cgit/newlib-cygwin/tree/winsup/cygwin/ker...
cryptonector 2y ago
Starting with main() instead of wmain() is enough to make it vulnerable :(
- kazinator 2y ago
  It's enough to make it suspicious, but there is a way for a C compiler on Windows to support regular main startup such that it is not vulnerable. It depends on which/whose piece of code takes the WCHAR-based command line and produces the char-based argv[]. Does that code do the "WorstFit" thing, or not. In my program, I could fix it (if necessary) by taking the WCHAR command line and parsing it myself, using my own UTF8 encoding routines to make the multi-byte strings.
  - cryptonector 2y ago
    Yes, that's what I'd do: write a `wmain()` that converts UTF-16 arguments to UTF-8 then calls the real `main()` (which you'll have to rename).

JimmyWilliams1 2y ago

[dead]

garganzol 2y ago

Microsoft was aware of this issue at least 1 year ago. I know this because they released a special code analysis rule CA2101 [1] that explicitly discouraged the use of the best-fit mapping. They mentioned security vulnerabilities in the rule’s description, but they were purposefully vague in details though. [1] https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2101 https://learn.microsoft.com/en-us/dotnet/fundamentals/code-a...

sharpshadow 2y ago

That’s amazing great read. According to for example this[0] post it’s possible to change code pages in windows in various ways and would allow the use of multiple BestFit scenarios on the same OS without reboot. Even combining them should be possible.

Aachen 2y ago
(you forgot the link)
- sharpshadow 2y ago
  Right. https://stackoverflow.com/a/56323296 https://stackoverflow.com/a/56323296

pornel 2y ago

It would be easily fixable if CommandlineToArgvA was obtaining the command line itself. Then instead of converting to ANSI and then parsing that, it could parse args in Unicode, and then convert argument by argument to ANSI. The output would be ANSI compatible, but split and unescaped in the true form. Unfortunately, the parsing is a two-step operation, with the application calling GetCommandLineA itself first and passing that to the parser, so a fix would need a hack to correlate the versions of the command line input without breaking when it's given a different string.

radarsat1 2y ago

Seems like a another possible fix would be to change the best fit mapping table to never generate control characters, but only alphanumerics. So map quote-like characters to 'q' and so on. This might be uglier and slightly change behaviour, but only for vulnerable applications.

account42 2y ago

Window's A APIs and conversion functions are best ingored entirely. Always use W functions and use your own converions (that can round-trip invalid UTF-16 like WTF-8) if you want to use an 8-bit encoding internally. Most (all?) of the exploits here are already bugs because the applications don't properly handle unicode data.