3 ms·
I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted),
by vbezhenar 10y ago
I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical strings. So while normalization might solve some simple problems, it's not a complete solution, so filesystem might just treat names as byte arrays and let user solve his problems.
- TazeTSchnitzel 10y agoConfusable characters look similar or the same to humans. Canonically equivalent Unicode sequences look the same to machines. The latter is a much more significant problem, because it can wreak havoc with interoperability.
- acdha 10y agoVery true - and it's amplified by inconsistency allowing problems to spread further before being noticed. At work I deal with a lot of Bag-It archives where we have a text manifest of checksums accompanying files on disk and this reliably bites users of tools when something (archive or network transfer tool, Git or SVN, transition to/from a Mac with HFS+, etc.) causes the encoding in the manifest not to match the local filesystem, and the confusion is amplified because some tools will handle normalization differences so the bug report is “why does tool A say this file is missing when Explorer/Finder and tool B say it's fine?”
- slrz 10y agoCanonically equivalent Unicode sequences look the same to machines. Memcmp disagrees, as do the default equality operators of most programming languages in existence.
- TazeTSchnitzel 10y agoSure, but normalisation can nonetheless happen automatically and implicitly in many places.
- vbezhenar 10y agoRust uses separate string type for file names. I think, that's a good approach. If language normalizes strings behind your back, that's not very good.
- deleted 10y ago[deleted]