4 ms·
Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings. > Windows was an early adopter of Unicod
by account42 2y ago
Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings.
> Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000
Wrong. Windows uses WTF-16 [0] despite what the documentation says.
[0] https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
- zombot 2y agoMicrosoft never implements a standard, they only ever implement their own shit. Sometimes it's a close enough parody of a standard to fool superficial onlookers, but that's as close as you'll ever get.
- formerly_proven 2y agoJava, NT, .NET, "wide" C and C++ and a few others from the same time frame ended up with WTF-16 because surrogate pairs didn't exist when they were designed. They were designed with UCS-2, which is a fixed-length encoding. Unicode 2.0 then extended that to be variable length (16/32-bit) using surrogate pairs and that's where all the systems come from which don't validate surrogate pairs.
- account42 2y ago"Wide" characters in C and C++ do not have a standardized width. On most non-Windows platforms they are actually four bytes so can cover the entirety of Unicode and then some in one wchar_t. Not that that's a more reasonable encoding for most applications than WTF-16.
- wbl 2y agoWhere did they think Cuneiform was going to fit? Even with unihan the BMP was getting consumed fast.
- formerly_proven 2y agoIn the beginning Unicode was only meant to be used for extant languages afaik.
- account42 2y agoYes, the clusterfuck that is CJK unification would not have been done if it was known that Unicode was going to go beyond 16-bit anyway.
- merb 2y agoIs that still true with dotnet? Especially since it has a surrogate pair api and a rune api? https://learn.microsoft.com/de-de/dotnet/api/system.char.issurrogatepair?view=net-9.0 https://learn.microsoft.com/de-de/dotnet/api/system.char.iss... https://learn.microsoft.com/en-us/dotnet/api/system.text.rune?view=net-9.0 https://learn.microsoft.com/en-us/dotnet/api/system.text.run...
- afiori 2y agoSurrogate pairs existed then, they just ignored them
- FirmwareBurner 2y ago>Microsoft never implements a standard Win32 ?
- p_ing 2y agoOp likely meant other's standards, not ones they create themselves. Sort of like AD, even though AD is objectively better than the mishmash of Kerb + LDAP + policy implementations out there.
- account42 2y agoThey don't even implement that "standard" correctly according to their own documentation.
- nitwit005 2y agoIn this case, they correctly met a standard, and the standard changed. If you look at OS-X there are similar issues. The Apple File System is case insensitive for a particular Unicode version.
- chrismorgan 2y agoNothing uses well-formed UTF-16. I don’t think I know of a single piece of software or library that uses 16-bit code units that validates them. In practice, “UTF-16” means “potentially-ill-formed UTF-16”. It’s that simple.
- ripe 2y agoThank you for posting the WTF-16 document. Very relevant for OP. It's an old problem that people on different OS's need to access the same filesystem, particularly Windows clients versus UNIXy clients. While UNIX filesystems traditionally accept any sequence of bytes except slash and NUL, and treat "." and ".." specially, Windows filesystems have had many additional restrictions on valid filenames, e.g., a list of "reserved names" that should not be used, short and long names, and case-insensitive names [1]. The NetApp filer, a NAS storage appliance, runs a specialized OS called Data ONTAP, which maintains a special on-disk representation called WAFL to enable this. WAFL is famous for its copy-on-write representation for file contents. [3] But in practice, most people in the real world are affected by its treatment of filenames. It's useful to take a look at how it solves these problems. The NAS presents the same set of files both as an NFS volume for UNIXy clients, as well as a CIFS volume for Windows clients. It does this by enhancing the directory entries with additional information and configuration features to satisfy both requirements. For typical problems and what features they offer to solve them, see their documentation page about naming files and paths [2]. [1] https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file https://learn.microsoft.com/en-us/windows/win32/fileio/namin... [2] https://docs.netapp.com/us-en/ontap/nfs-admin/multi-byte-file-directory-qtree-names-concept.html https://docs.netapp.com/us-en/ontap/nfs-admin/multi-byte-fil... [3] Dave Hitz, "File System Design for an NFS File Server Appliance", https://www.cs.princeton.edu/courses/archive/fall17/cos318/readings/netapp.pdf https://www.cs.princeton.edu/courses/archive/fall17/cos318/r...
- p_ing 2y agoNTFS has the same two restrictions that many UNIX file systems have, NUL and slash. The APIs, Win32 in the case of [1], have further restrictions. If you want you can use a different API/personality and write whatever value you'd like (sans NUL and /) provided said personality has no limits -- NTFS does no validation itself. In practice, Win32 being the default personality makes said plethlora of restrictions true, but it isn't a "filesystem" limitation, rather an API restriction. A nuanced if unimportant difference.
- sumtechguy 2y agoIf you want to subtly break a windows install you can use setCaseSensitiveInfo with fsutil. It turns on/off case sensitivity for a directory. There is also a similar set of options for samba shares which comes with interesting tradeoffs for speed of reading a directory list.
- layer8 2y agoHistorically, this is because Windows NT used UCS-2 [0]. Unicode only moved to beyond 65536 characters, and introduced the concept of surrogate pairs, with Unicode 2.0 in 1996. [0] https://www.unicode.org/faq/utf_bom.html#utf16-11 https://www.unicode.org/faq/utf_bom.html#utf16-11