7 ms·

Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

85 points by feldrim 2y ago

rob74 2y ago
Or otherwise said: Surrogate Pairs are used in UTF-16 (which uses two bytes per character, so it can encode up to 65536 characters) to encode Unicode characters that have code points that can't be encoded using just two bytes.
- feldrim 2y ago
  Yep. The quirk here is that the surrogates, that are merely enablers for other characters, can be paired with each other. With the absence of other valid characters, they are not enabling anything. One assumes there is a validation but it does not exist here.
  - layer8 2y ago
    There is no validation on the file system level because file names in NTFS are sequences of arbitrary 16-bit values, similar to how on Unix file systems, file names are sequences of arbitrary 8-bit values. Arguably the situation on Unix is worse, because there the interpretation and validity depends on the current locale.
    - feldrim 2y ago
      Totally. These are design choices made by development teams. But as users, we "assume" all are readable until one day we learn that it does not work that way. Until I came across this issue, I assumed them to be all valid, renderable characters.
n_plus_1_acc 2y ago
I think it's hilarious that the event viewer XML gets borked.
- feldrim 2y ago
  I am not 100% sure but mmc.exe has not been updated for years and it must be relying on WebBrowser control of Internet Explorer. Yes, IE is still alive in Windows. https://learn.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa752040(v=vs.85) https://learn.microsoft.com/en-us/previous-versions/windows/...
  - account42 2y ago
    And we should all be thankful for that. Just imagine if all those system tools were as "useful" as the modernized windows settings.
kzrdude 2y ago
I remember that in Mac OS X times, sometime between OS X v10.1 and 10.4, a system upgrade caused a bunch of unicode named files to become inaccessible/untouchable (but still present with a directory listing). At the time I didn't have the skills to figure out what had happened. I'm still curious to know if it was an intended breaking change.
- kps 2y ago
  OS X (at the POSIX level) assumes UTF-8 and normalizes file names to decomposed form (NFD). If for example you `date >$'\xC3\xBC'` (i.e. ‘ü’), then the actual stored file name is `$'\x75\xCC\88'` (i.e. ‘ü’ — assuming HN or my browser don't normalize!) and both `cat $'\xC3\xBC'` and `cat $'\x75\xCC\88'` (or ‘ü’ or ‘ü’) both work.
  - kzrdude 2y ago
    Unrelated to my question above, I think
qingcharles 2y ago
Aha! Found the name of my next album. Try downloading me on Napster now!
theiebrjfb 2y ago
Yet another reason to use Linux everywhere. It is 2025 and Windows (and probably Mac) users have to deal with weird Unicode filesystem issues. Good luck puting Chinese characters or emoticons into filenames. Ext4 filename has maximal length 255 characters. That is the only legacy limit you have to deal with as a Linux user. And even that can be avoided by using more modern filesystems. And we get filesystem level snapshots etc...
- layer8 2y ago
  You have the same, if not worse, issue on Linux with filenames that aren’t valid UTF-8 sequences. Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS.
  - kwertzzz 2y ago
    > Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS. If you change the locale to an uninstalled one, then yes. But if the locale is installed, then I don't see a problem. echo $LANG # output: en_US.UTF-8 touch fusée.txt LANG=fr_FR.UTF-8 ls # output: 'fus'$'\303\251''e.txt' sudo locale-gen fr_FR.UTF-8 sudo update-locale LANG=fr_FR.UTF-8 ls # output: fusée.txt Are you maybe using non-UTF-8 locale?
    - layer8 2y ago
      Yes, I mean locales like fr_FR.ISO-8859-15, ja_JP.SJIS or zh_CN.GBK. While these probably aren’t used much anymore, it still means that your filenames can break just by setting an environment variable. Or issues like here: https://news.ycombinator.com/item?id=16992546 https://news.ycombinator.com/item?id=16992546
- feldrim 2y ago
  I see two points here. First, you did not read the article and did not see the footnote that these are valid in Linux as well. Second, your comment shows you are lacking the knowledge on Linux as well. In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”). Other than that, all characters are valid paths. If you consider these a problem, I'd like to remind that the 2048 surrogate pairs is a really small subset of unrenderable combinations allowed in Linux. Anyone are free to have their opinions but at least, before making bold claims, please do your due diligence.
Devasta 2y ago
Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.
- rurban 2y ago
  No, the real solution is to follow the unicode security guidelines for identifiers. Esp. on linux, where the silly garbage-in, garbage-out mantra doesn't fly with identifiers, because identifiers need to stay identifiable. Apple HPFS did some things right. They did at least NFD. But linux insanities brought them back to -Whomoglyph attacks
- wheybags 2y ago
  I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing becomes super easy like in the old ascii only k&r c example code. I'm not 100% certain, but I don't think there's a widely used language that couldn't be represented by that. Of course, you lose round trip ability with legacy encodings, which is why we have the mess that is unicode. Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Oh well, it's too late now so I guess we just accept it.
  - jerf 2y ago
    For better or worse, thanks to emoji Zero Width Joiner support [1], we're well on our way to there being more than 4 billion potential Unicode "characters". 4 billion is only 32 bits and you start spending a few bits here on hair style and a few bits there on skin color and a few bits on "misc" and then allow arbitrary combinations of them into composite families [2] and you can burn through 32-bits fairly quickly. I don't think we're there yet. I think if someone did make a complete list of "valid" emoji right now, which for the sake of argument I'll call "formally defined in the Unicode standard", it would even on an absolute scale look like we're a long ways away from a full 32-bits of valid combinations. But you have to think of this on the log scale because this is about "bits" and those four-person families are already quite a long ways along to a full 32 bits. It wouldn't take much more customization, or the formal addition of more people in a group, to get there. And someone who knows more about Unicode than I do may be able to establish that there are already in the standard ways to get to more than 32 bits' worth of data in a single standardized glyph; I certainly wouldn't bet much against that already being true. (Personally, I'll go with "worse". In hindsight, we should probably have frozen Unicode into the original Docomo (and the other phone company that had them) emoji necessary for interoperability, and then created the emoji as an extension into Unicode. It seems like it would be useful to "support Unicode" without having to come with the complete understanding of what is increasingly the most complicated "language" in Unicode; forget doing good Arabic rendering or trying to understand an ideographic language, the emojis blow all that complexity away now. But here we are.) [1]: https://unicode.org/emoji/charts/emoji-zwj-sequences.html https://unicode.org/emoji/charts/emoji-zwj-sequences.html [2]: https://www.unicode.org/reports/tr51/#Multi_Person_Groupings https://www.unicode.org/reports/tr51/#Multi_Person_Groupings
feldrim 2y ago
Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions made behind these. If anyone has information, please let me know.
- formerly_proven 2y ago
  The Linux section just seems to be artifacts of the WSL hacks, it has nothing to do with how Linux filenames function. Those are simply bags of bytes, the encoding only matters for displaying them, and isn't interpreted internally. ls failing to access the .exe is clearly a WSL filesystem issue and not a Linux / ls issue. You also can't set a UTF-16 locale because that's not what a locale is. UTF-16/32 vs SBCS and UTF-8 is the wide/narrow character distinction, which is a whole separate thing, different ABIs, different APIs.
  - feldrim 2y ago
    WSL-to-Windows, yes, it is due to translations. But within the WSL, not sure. I'll try to replicate them on a Ubuntu VM for comparison.
- zombot 2y ago
  WSL is not Linux, despite whatever Microsoft says.
  - p_ing 2y ago
    WSL is Linux -- it's an automatically managed VM with some special sauce for connectivity between the parent partition and guest.
    - anonymfus 2y ago
      WSL2 is what you describe, WSL1 is not.
somewhereoutth 2y ago
Ah got caught by surrogate pairs recently:- javascript sees them as 2 chars when e.g. slicing strings, so it is possible to end up with invalid strings if you chop between a pair.
dwdz 2y ago
The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names. ls -l win32/ total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\251\205''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\255\223''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\272\257''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\264\207''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\261\246''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\254\266''.exe' ...
- feldrim 2y ago
  Oh, great. Can you also share the locale? I'll write another Postscriptum section then.
  - dwdz 2y ago
    LANG=en_IE.UTF-8 LANGUAGE=en_IE:en LC_CTYPE="en_IE.UTF-8" LC_NUMERIC="en_IE.UTF-8" LC_TIME="en_IE.UTF-8" LC_COLLATE="en_IE.UTF-8" LC_MONETARY="en_IE.UTF-8" LC_MESSAGES="en_IE.UTF-8" LC_PAPER="en_IE.UTF-8" LC_NAME="en_IE.UTF-8" LC_ADDRESS="en_IE.UTF-8" LC_TELEPHONE="en_IE.UTF-8" LC_MEASUREMENT="en_IE.UTF-8" LC_IDENTIFICATION="en_IE.UTF-8" LC_ALL=
    - rstuart4133 2y ago
      Not the OP, but have you tried LANG=C and getting rid of the rest?
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
mofeien 2y ago
Hi, thanks for the interesting submission! I was a bit confused by the detour via utf-8 to arrive at the code points and had to look up UTF-8 encoding first to understand how they relate. Then I tried out the following candidate = chr(0xD800) candidate2 = bytes([0xED, 0xA0, 0x80]).decode('utf-8', errors='surrogatepass') print(candidate == candidate2) # True and it seems that you could just iterate over code points directly with the `chr()` function.
- feldrim 2y ago
  I f I remember correctly, I tried that but in order to cover the exact range I need, the high and low surrogates, I picked this way out of practicality. It was just easier.
account42 2y ago
Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
- zombot 2y ago
  Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.
  - formerly_proven 2y ago
    Java, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.
    - account42 2y ago
      "Wide" characters in C and C++ do not have a standardized width. On most non-Windows platforms they are actually four bytes so can cover the entirety of Unicode and then some in one wchar_t. Not that that's a more reasonable encoding for most applications than WTF-16.
    - wbl 2y ago
      Where did they think Cuneiform was going to fit? Even with unihan the BMP was getting consumed fast.
      - formerly_proven 2y ago
ooterness 2y ago
Why does the Windows filesystem allow filenames with invalid strings? It seems obvious that attempts to create files with such filenames ought to be blocked.
- feldrim 2y ago
  It's mentioned in a comment here that the existing restrictions are due to Windows APIs and NTFS does not check file names in a restricted way. Therefore, if devs want to filter these out or not in the API, is another story.
ge96 2y ago
I remember a long time ago I accidentally put some symbol in a folder name like ? in Windows had problems

85 points by feldrim 2y ago

Or otherwise said: Surrogate Pairs are used in UTF-16 (which uses two bytes per character, so it can encode up to 65536 characters) to encode Unicode characters that have code points that can't be encoded using just two bytes.

feldrim 2y ago
Yep. The quirk here is that the surrogates, that are merely enablers for other characters, can be paired with each other. With the absence of other valid characters, they are not enabling anything. One assumes there is a validation but it does not exist here.
- layer8 2y ago
  There is no validation on the file system level because file names in NTFS are sequences of arbitrary 16-bit values, similar to how on Unix file systems, file names are sequences of arbitrary 8-bit values. Arguably the situation on Unix is worse, because there the interpretation and validity depends on the current locale.
  - feldrim 2y ago
    Totally. These are design choices made by development teams. But as users, we "assume" all are readable until one day we learn that it does not work that way. Until I came across this issue, I assumed them to be all valid, renderable characters.

n_plus_1_acc 2y ago

I think it's hilarious that the event viewer XML gets borked.

feldrim 2y ago
I am not 100% sure but mmc.exe has not been updated for years and it must be relying on WebBrowser control of Internet Explorer. Yes, IE is still alive in Windows. https://learn.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa752040(v=vs.85) https://learn.microsoft.com/en-us/previous-versions/windows/...
- account42 2y ago
  And we should all be thankful for that. Just imagine if all those system tools were as "useful" as the modernized windows settings.

kzrdude 2y ago

I remember that in Mac OS X times, sometime between OS X v10.1 and 10.4, a system upgrade caused a bunch of unicode named files to become inaccessible/untouchable (but still present with a directory listing). At the time I didn't have the skills to figure out what had happened. I'm still curious to know if it was an intended breaking change.

kps 2y ago
OS X (at the POSIX level) assumes UTF-8 and normalizes file names to decomposed form (NFD). If for example you `date >$'\xC3\xBC'` (i.e. ‘ü’), then the actual stored file name is `$'\x75\xCC\88'` (i.e. ‘ü’ — assuming HN or my browser don't normalize!) and both `cat $'\xC3\xBC'` and `cat $'\x75\xCC\88'` (or ‘ü’ or ‘ü’) both work.
- kzrdude 2y ago
  Unrelated to my question above, I think

qingcharles 2y ago

Aha! Found the name of my next album. Try downloading me on Napster now!

theiebrjfb 2y ago

Yet another reason to use Linux everywhere. It is 2025 and Windows (and probably Mac) users have to deal with weird Unicode filesystem issues. Good luck puting Chinese characters or emoticons into filenames. Ext4 filename has maximal length 255 characters. That is the only legacy limit you have to deal with as a Linux user. And even that can be avoided by using more modern filesystems. And we get filesystem level snapshots etc...

layer8 2y ago
You have the same, if not worse, issue on Linux with filenames that aren’t valid UTF-8 sequences. Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS.
- kwertzzz 2y ago
  > Not to mention that on Linux switching the locale may change the interpretation of filenames as characters, which isn’t the case with NTFS. If you change the locale to an uninstalled one, then yes. But if the locale is installed, then I don't see a problem. echo $LANG # output: en_US.UTF-8 touch fusée.txt LANG=fr_FR.UTF-8 ls # output: 'fus'$'\303\251''e.txt' sudo locale-gen fr_FR.UTF-8 sudo update-locale LANG=fr_FR.UTF-8 ls # output: fusée.txt Are you maybe using non-UTF-8 locale?
  - layer8 2y ago
    Yes, I mean locales like fr_FR.ISO-8859-15, ja_JP.SJIS or zh_CN.GBK. While these probably aren’t used much anymore, it still means that your filenames can break just by setting an environment variable. Or issues like here: https://news.ycombinator.com/item?id=16992546 https://news.ycombinator.com/item?id=16992546
feldrim 2y ago
I see two points here. First, you did not read the article and did not see the footnote that these are valid in Linux as well. Second, your comment shows you are lacking the knowledge on Linux as well. In Linux, as I have written in the foot note, accepts anything but 0x00 (null) and 0x2F (“/”). Other than that, all characters are valid paths. If you consider these a problem, I'd like to remind that the 2048 surrogate pairs is a really small subset of unrenderable combinations allowed in Linux. Anyone are free to have their opinions but at least, before making bold claims, please do your due diligence.

Devasta 2y ago

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake. The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

rurban 2y ago
No, the real solution is to follow the unicode security guidelines for identifiers. Esp. on linux, where the silly garbage-in, garbage-out mantra doesn't fly with identifiers, because identifiers need to stay identifiable. Apple HPFS did some things right. They did at least NFD. But linux insanities brought them back to -Whomoglyph attacks
wheybags 2y ago
I still think we should have forced everything into a 32-bit char, with no distinction between codepoints and grapheme clusters. One press on backspace removes one char. Address of char 7 is base+7x4. String length is byte length x 4. cat /dev/urandom is a valid string, it's the font's job to deal with unknown byte values, if you just want to process the text you dont need to care. Everything about text processing becomes super easy like in the old ascii only k&r c example code. I'm not 100% certain, but I don't think there's a widely used language that couldn't be represented by that. Of course, you lose round trip ability with legacy encodings, which is why we have the mess that is unicode. Oh and silly things like unicode flag emojis wouldn't work, but honestly maybe that would be for the best. Oh well, it's too late now so I guess we just accept it.
- jerf 2y ago
  For better or worse, thanks to emoji Zero Width Joiner support [1], we're well on our way to there being more than 4 billion potential Unicode "characters". 4 billion is only 32 bits and you start spending a few bits here on hair style and a few bits there on skin color and a few bits on "misc" and then allow arbitrary combinations of them into composite families [2] and you can burn through 32-bits fairly quickly. I don't think we're there yet. I think if someone did make a complete list of "valid" emoji right now, which for the sake of argument I'll call "formally defined in the Unicode standard", it would even on an absolute scale look like we're a long ways away from a full 32-bits of valid combinations. But you have to think of this on the log scale because this is about "bits" and those four-person families are already quite a long ways along to a full 32 bits. It wouldn't take much more customization, or the formal addition of more people in a group, to get there. And someone who knows more about Unicode than I do may be able to establish that there are already in the standard ways to get to more than 32 bits' worth of data in a single standardized glyph; I certainly wouldn't bet much against that already being true. (Personally, I'll go with "worse". In hindsight, we should probably have frozen Unicode into the original Docomo (and the other phone company that had them) emoji necessary for interoperability, and then created the emoji as an extension into Unicode. It seems like it would be useful to "support Unicode" without having to come with the complete understanding of what is increasingly the most complicated "language" in Unicode; forget doing good Arabic rendering or trying to understand an ideographic language, the emojis blow all that complexity away now. But here we are.) [1]: https://unicode.org/emoji/charts/emoji-zwj-sequences.html https://unicode.org/emoji/charts/emoji-zwj-sequences.html [2]: https://www.unicode.org/reports/tr51/#Multi_Person_Groupings https://www.unicode.org/reports/tr51/#Multi_Person_Groupings

feldrim 2y ago

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names: 1. On Windows, accessed by WSL 2. On Linux (WSL), using UTF-8 locale 3. On Linux (WSL), using POSIX locale The difference is weird for me as a user. I'd like to know about the decisions made behind these. If anyone has information, please let me know.

formerly_proven 2y ago
The Linux section just seems to be artifacts of the WSL hacks, it has nothing to do with how Linux filenames function. Those are simply bags of bytes, the encoding only matters for displaying them, and isn't interpreted internally. ls failing to access the .exe is clearly a WSL filesystem issue and not a Linux / ls issue. You also can't set a UTF-16 locale because that's not what a locale is. UTF-16/32 vs SBCS and UTF-8 is the wide/narrow character distinction, which is a whole separate thing, different ABIs, different APIs.
- feldrim 2y ago
  WSL-to-Windows, yes, it is due to translations. But within the WSL, not sure. I'll try to replicate them on a Ubuntu VM for comparison.
zombot 2y ago
WSL is not Linux, despite whatever Microsoft says.
- p_ing 2y ago
  WSL is Linux -- it's an automatically managed VM with some special sauce for connectivity between the parent partition and guest.
  - anonymfus 2y ago
    WSL2 is what you describe, WSL1 is not.

somewhereoutth 2y ago

Ah got caught by surrogate pairs recently:- javascript sees them as 2 chars when e.g. slicing strings, so it is possible to end up with invalid strings if you chop between a pair.

dwdz 2y ago

The script works just fine on real Linux, it creates 2048 files and ls command lists them all with different names. ls -l win32/ total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\277\237''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\267\213''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\240\220''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\274\273''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\251\205''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\255\223''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\272\257''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\264\207''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\261\246''.exe' -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 ''$'\355\254\266''.exe' ...

feldrim 2y ago
Oh, great. Can you also share the locale? I'll write another Postscriptum section then.
- dwdz 2y ago
  LANG=en_IE.UTF-8 LANGUAGE=en_IE:en LC_CTYPE="en_IE.UTF-8" LC_NUMERIC="en_IE.UTF-8" LC_TIME="en_IE.UTF-8" LC_COLLATE="en_IE.UTF-8" LC_MONETARY="en_IE.UTF-8" LC_MESSAGES="en_IE.UTF-8" LC_PAPER="en_IE.UTF-8" LC_NAME="en_IE.UTF-8" LC_ADDRESS="en_IE.UTF-8" LC_TELEPHONE="en_IE.UTF-8" LC_MEASUREMENT="en_IE.UTF-8" LC_IDENTIFICATION="en_IE.UTF-8" LC_ALL=
  - rstuart4133 2y ago
    Not the OP, but have you tried LANG=C and getting rid of the rest?
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

mofeien 2y ago

Hi, thanks for the interesting submission! I was a bit confused by the detour via utf-8 to arrive at the code points and had to look up UTF-8 encoding first to understand how they relate. Then I tried out the following candidate = chr(0xD800) candidate2 = bytes([0xED, 0xA0, 0x80]).decode('utf-8', errors='surrogatepass') print(candidate == candidate2) # True and it seems that you could just iterate over code points directly with the `chr()` function.

feldrim 2y ago
I f I remember correctly, I tried that but in order to cover the exact range I need, the high and low surrogates, I picked this way out of practicality. It was just easier.

account42 2y ago

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000 Wrong. Windows uses WTF-16 [0] despite what the documentation says. [0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

zombot 2y ago
Microsoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.
- formerly_proven 2y ago
  Java, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.
  - account42 2y ago
    "Wide" characters in C and C++ do not have a standardized width. On most non-Windows platforms they are actually four bytes so can cover the entirety of Unicode and then some in one wchar_t. Not that that's a more reasonable encoding for most applications than WTF-16.
  - wbl 2y ago
    Where did they think Cuneiform was going to fit? Even with unihan the BMP was getting consumed fast.
    - formerly_proven 2y ago

ooterness 2y ago

Why does the Windows filesystem allow filenames with invalid strings? It seems obvious that attempts to create files with such filenames ought to be blocked.

feldrim 2y ago
It's mentioned in a comment here that the existing restrictions are due to Windows APIs and NTFS does not check file names in a restricted way. Therefore, if devs want to filter these out or not in the API, is another story.

ge96 2y ago

I remember a long time ago I accidentally put some symbol in a folder name like ? in Windows had problems