3 ms·

It's only inert if you're not transforming it (for example, calling toLower on all strings passed in) or doing equality checks. Many applications use these and

by JTbane 4y ago

It's only inert if you're not transforming it (for example, calling toLower on all strings passed in) or doing equality checks. Many applications use these and it is a minefield in Unicode.

tialaramex 4y ago
But what's the semantic purpose of these operations? Linguistically these are very strange things to do to arbitrary text, and yet because in ASCII it was easy, programmers got into the habit of just doing it all over the place without thinking about what (if anything) it meant. For transform operations like "toLower" often it turns out that to the extent it means anything it's only relevant to ASCII and so an operation which only transforms ASCII is appropriate anyway.
- throwaway09223 4y ago
  > But what's the semantic purpose of these operations? The list is infinite, but let's use a very simple one which is the focus of this article: Names of things. You name a file on a case insensitive filesystem. Should the case insensitivity only apply to certain encodings, or all of them? You'll need to do a case transformation and string comparison to make this work. You create an email address, per rfc6531. You'll need to deal with case flattening, string comparisons, and lexical parsing. These are not strange operations. These are not only common they are obviously required operations, because names identify things and names (eg emails) are often structured and must be parsed and understood. Can you imagine the fallout if software developers said "sorry, you can only search for ASCII names because we think indexing unicode strings is a very strange thing to do?"
  - tialaramex 4y ago
    > You'll need to do a case transformation and string comparison to make this work. Better to say you need a canonicalisation table and you will need to canonicalise the text. NTFS filesystems actually carry their canonicalisation table with them, so if you happen to be using a USB drive with an NTFS filesystem where Z and Q are just different versions of the same hypothetical "Q-Z" character, filenames ZZ150 and QQ150 are indistinguishable on that USB drive, even though your Windows PC has no trouble distinguishing Z from Q. Notice that unlike "case transformation" this doesn't pretend we're doing something with larger semantic value, and indeed the actual names on disk aren't promised to be Unicode text under NTFS, they're just arbitrary 16-bit values which probably make sense as UCS-2 or these days UTF-16 when rendered, the tables just convert some values into others.
    - throwaway09223 4y ago
      Do you understand that filesystem drivers are yet another example of places where software developers need to do these types of operations which you're dismissing as "very strange things?" How do you propose that I, a person who among other things develops filesystems, implement a canonicalization table without doing these "very strange things?" Furthermore, filesystem operations are not limited to calls into the filesystem driver. Programs often need to deal with paths outside of filesystem calls: The names of files need to be displayed on screens. They need to be indexed in databases and searched for. They need to be encapsulated within other protocols and data formats and often need to be parsed and understood. You are essentially agreeing with me that these are normal, necessary things. Your position appears to be "someone else does that type of thing for me," which is not a very compelling argument.
      - GoblinSlayer 4y ago
        Case insensitive file systems might as well be a historic accident, because lower case letters didn't always exist and then it was kept for backward compatibility.
        throwaway09223 4y ago
        Case insensitive functions are common. Filesystems are just one of many places where this kind of function is expected. It's just a particularly common canonicalization function. Canonicalization is a common need across all languages as we've already clearly established.
  - GoblinSlayer 4y ago
    I think you shouldn't try to reproduce the filesystem name comparison algorithm. Check if file exists or generate numeric name. I can't find where rfc6531 requires case flattening. The word "case" is used there in the meaning of "situation".
    - throwaway09223 4y ago
      Case sensitivity is addressed in other RFCs relating to email and DNS. > Check if file exists or generate numeric name. Nonsense. You may not be in control of the names, and checking a file with stat(2) may not address the problem at hand (and: is many orders of magnitude different in cost as well).
GoblinSlayer 4y ago
I imagine those applications have a better idea what sanitation they need. Is it even the same for all of them? And toLower is what they do now, but how do you predict the algorithm won't change in future? When you sanitize proactively for future algorithms, it's just data loss.

by JTbane 4y ago

It's only inert if you're not transforming it (for example, calling toLower on all strings passed in) or doing equality checks. Many applications use these and it is a minefield in Unicode.

tialaramex 4y ago

But what's the semantic purpose of these operations? Linguistically these are very strange things to do to arbitrary text, and yet because in ASCII it was easy, programmers got into the habit of just doing it all over the place without thinking about what (if anything) it meant. For transform operations like "toLower" often it turns out that to the extent it means anything it's only relevant to ASCII and so an operation which only transforms ASCII is appropriate anyway.

throwaway09223 4y ago
> But what's the semantic purpose of these operations? The list is infinite, but let's use a very simple one which is the focus of this article: Names of things. You name a file on a case insensitive filesystem. Should the case insensitivity only apply to certain encodings, or all of them? You'll need to do a case transformation and string comparison to make this work. You create an email address, per rfc6531. You'll need to deal with case flattening, string comparisons, and lexical parsing. These are not strange operations. These are not only common they are obviously required operations, because names identify things and names (eg emails) are often structured and must be parsed and understood. Can you imagine the fallout if software developers said "sorry, you can only search for ASCII names because we think indexing unicode strings is a very strange thing to do?"
- tialaramex 4y ago
  > You'll need to do a case transformation and string comparison to make this work. Better to say you need a canonicalisation table and you will need to canonicalise the text. NTFS filesystems actually carry their canonicalisation table with them, so if you happen to be using a USB drive with an NTFS filesystem where Z and Q are just different versions of the same hypothetical "Q-Z" character, filenames ZZ150 and QQ150 are indistinguishable on that USB drive, even though your Windows PC has no trouble distinguishing Z from Q. Notice that unlike "case transformation" this doesn't pretend we're doing something with larger semantic value, and indeed the actual names on disk aren't promised to be Unicode text under NTFS, they're just arbitrary 16-bit values which probably make sense as UCS-2 or these days UTF-16 when rendered, the tables just convert some values into others.
  - throwaway09223 4y ago
    Do you understand that filesystem drivers are yet another example of places where software developers need to do these types of operations which you're dismissing as "very strange things?" How do you propose that I, a person who among other things develops filesystems, implement a canonicalization table without doing these "very strange things?" Furthermore, filesystem operations are not limited to calls into the filesystem driver. Programs often need to deal with paths outside of filesystem calls: The names of files need to be displayed on screens. They need to be indexed in databases and searched for. They need to be encapsulated within other protocols and data formats and often need to be parsed and understood. You are essentially agreeing with me that these are normal, necessary things. Your position appears to be "someone else does that type of thing for me," which is not a very compelling argument.
    - GoblinSlayer 4y ago
      Case insensitive file systems might as well be a historic accident, because lower case letters didn't always exist and then it was kept for backward compatibility.
      - throwaway09223 4y ago
        Case insensitive functions are common. Filesystems are just one of many places where this kind of function is expected. It's just a particularly common canonicalization function. Canonicalization is a common need across all languages as we've already clearly established.
- GoblinSlayer 4y ago
  I think you shouldn't try to reproduce the filesystem name comparison algorithm. Check if file exists or generate numeric name. I can't find where rfc6531 requires case flattening. The word "case" is used there in the meaning of "situation".
  - throwaway09223 4y ago
    Case sensitivity is addressed in other RFCs relating to email and DNS. > Check if file exists or generate numeric name. Nonsense. You may not be in control of the names, and checking a file with stat(2) may not address the problem at hand (and: is many orders of magnitude different in cost as well).

GoblinSlayer 4y ago

I imagine those applications have a better idea what sanitation they need. Is it even the same for all of them? And toLower is what they do now, but how do you predict the algorithm won't change in future? When you sanitize proactively for future algorithms, it's just data loss.