4 ms·

just for giggles: http://www.unicode.org/Public/3.1-Update1/CaseFolding-4.txt http://www.unicode.org/Public/3.1-Update1/CaseFolding-4.txt

fanf2 2y ago
Heh, thank goodness I don’t have to deal with all that! This code is ascii-only because it arose from working on the DNS. There are other protocols that are ascii-case-insensitive so it turns up a lot in the hot path of many servers.
- tialaramex 2y ago
  This is presumably Rust's u8::to_ascii_lowercase rather than C's tolower since tolower is locale sensitive (which you don't care about) and has Undefined Behaviour (because of course it does it's a C function and who cares about correctness) Or possibly u8::make_ascii_lowercase which is the same function but with in-place mutation.
Asooka 2y ago
There is a difference between strings used internally, usually as IDs, and text entered by a human. For the former you'd always use straight ASCII in 8-bit encoding, for the latter ... things get difficult. A straightforward example are DNS addresses - they can technically contain almost any Unicode, but that is always converted to a very limited subset of ASCII for actual DNS resolution, which in turn is case-insensitive. There are of course things like programming languages with case-insensitive identifiers that support all human writing systems in Unicode. If that's what you're dealing with, you have my condolences.
- mananaysiempre 2y ago
  For what it’s worth, with IDNs you’re still going to do a kind of case folding using stringprep before doing the query, and that isn’t really better than the table GP linked. ASCII-only case-insensitive matching is indeed a thing, but it’s usually mutually exclusive with (non-programmer) user-entered data.
- tialaramex 2y ago
  On the wire DNS questions and answers are case preserving but not case sensitive which is important for correctness. DNS was designed a long time ago and doesn't have enough places to hide randomness (~30 bits at most are available) to protect it against a realistic attacker, so, for most names we do a terrible trick in the real world - we randomize the case during querying. This is called DNS-0x20 for obvious reasons. Suppose a web browser wants to know news.ycombinator.com AAAA? but bad guys know it's about to ask this (e.g. they use JS to force that lookup when they wanted), they can shove a billion bogus answers (one for every possible random value) onto the wire and have a great chance to trick the browser into accepting one of these answers which seemingly is to the question it just asked. But, if we instead pick random cases we're asking about, say, NeWS.yCOmbinAtOR.cOM and we can ignore answers for nEWS.yCOMBINATOR.cOM or news.ycombinator.com or NEWS.YCOMBINATOR.COM or any other spelling. Bad guys now need to do many orders of magnitude more expensive work for the same results.
- deleted 2y ago
  [deleted]
- account42 2y ago
  > There are of course things like programming languages with case-insensitive identifiers that support all human writing systems in Unicode. If that's what you're dealing with, you have my condolences. Fun times when an upgrade of the Unicode library used by your compiler changes the semantics of you program.
- hnben 2y ago
  > For the former you'd always use straight ASCII in 8-bit encoding why is that? IMO internal strings should be treated as black-box byte-arrays, i.e. the specific content does not matter to the software, except for checking equality between to strings. In this case it should not matter to the software if it is unicode or whatever.
atoav 2y ago
Note for the first example which transforms the German maße to MASSE that the German language has an uppercase Eszett as well: ẞ This is as of now not widely deployed and few fonts support it, but in theory it is there now.
pdpi 2y ago
While we're having fun with that: # Capitalising an eszett changes the string length. >>> "straße".upper() 'STRASSE' # If you don't specify the locale, round-trip upper/lower case # messes up the dotless i used in turkic languages. >>> 'ı'.upper().lower() 'i'
- kleiba 2y ago
  Up until a few years ago, the first conversion would have been the official way to write a German word that contains the ligature ß (a lower-case letter) in all caps, because there was no corresponding upper-case letter. However, in 2017, an upper-case variant [1] was introduced into the official German orthography, so the conversion cited here should no longer be necessary. [1] https://en.wikipedia.org/wiki/Capital_%E1%BA%9E https://en.wikipedia.org/wiki/Capital_%E1%BA%9E
  - Tomte 2y ago
    It is extremely unusual, and the addition to Unicode was very controversial. Basically, some type designers liked to play with capital ß and thought it cool to have it included into Unicode. There was a whole campaign for inclusion, and it was a big mistake. Because even though existing use must be shown to merit inclusion, they only managed to find a single book (a dictionary) printed in more than a few copies. From East Germany. And that book only used the capital ß for a single edition (out of dozens of editions) and reverted straight back to double s. Somehow, that still was enough for Unicode. Capital ß is a shit show, and it only confuses native speakers, because close to none of them have ever seen that glyph before. It has no real historic lineage (like the long s, for example, that pretty much nobody under 80 knows, either), it is a faux-retro modern design, an idle plaything.
    - kleiba 2y ago
      Well, what can you do... amtlich is amtlich. And the behavior of a function that converts ß to double upper-case S can certainly be discussed too, if only for the fact that a round-trip toUpper().toLower() will not give you back the original word.

fanf2 2y ago

Heh, thank goodness I don’t have to deal with all that! This code is ascii-only because it arose from working on the DNS. There are other protocols that are ascii-case-insensitive so it turns up a lot in the hot path of many servers.

tialaramex 2y ago
This is presumably Rust's u8::to_ascii_lowercase rather than C's tolower since tolower is locale sensitive (which you don't care about) and has Undefined Behaviour (because of course it does it's a C function and who cares about correctness) Or possibly u8::make_ascii_lowercase which is the same function but with in-place mutation.

Asooka 2y ago

There is a difference between strings used internally, usually as IDs, and text entered by a human. For the former you'd always use straight ASCII in 8-bit encoding, for the latter ... things get difficult. A straightforward example are DNS addresses - they can technically contain almost any Unicode, but that is always converted to a very limited subset of ASCII for actual DNS resolution, which in turn is case-insensitive. There are of course things like programming languages with case-insensitive identifiers that support all human writing systems in Unicode. If that's what you're dealing with, you have my condolences.

mananaysiempre 2y ago
For what it’s worth, with IDNs you’re still going to do a kind of case folding using stringprep before doing the query, and that isn’t really better than the table GP linked. ASCII-only case-insensitive matching is indeed a thing, but it’s usually mutually exclusive with (non-programmer) user-entered data.
tialaramex 2y ago
On the wire DNS questions and answers are case preserving but not case sensitive which is important for correctness. DNS was designed a long time ago and doesn't have enough places to hide randomness (~30 bits at most are available) to protect it against a realistic attacker, so, for most names we do a terrible trick in the real world - we randomize the case during querying. This is called DNS-0x20 for obvious reasons. Suppose a web browser wants to know news.ycombinator.com AAAA? but bad guys know it's about to ask this (e.g. they use JS to force that lookup when they wanted), they can shove a billion bogus answers (one for every possible random value) onto the wire and have a great chance to trick the browser into accepting one of these answers which seemingly is to the question it just asked. But, if we instead pick random cases we're asking about, say, NeWS.yCOmbinAtOR.cOM and we can ignore answers for nEWS.yCOMBINATOR.cOM or news.ycombinator.com or NEWS.YCOMBINATOR.COM or any other spelling. Bad guys now need to do many orders of magnitude more expensive work for the same results.
deleted 2y ago
[deleted]
account42 2y ago
> There are of course things like programming languages with case-insensitive identifiers that support all human writing systems in Unicode. If that's what you're dealing with, you have my condolences. Fun times when an upgrade of the Unicode library used by your compiler changes the semantics of you program.
hnben 2y ago
> For the former you'd always use straight ASCII in 8-bit encoding why is that? IMO internal strings should be treated as black-box byte-arrays, i.e. the specific content does not matter to the software, except for checking equality between to strings. In this case it should not matter to the software if it is unicode or whatever.

atoav 2y ago

Note for the first example which transforms the German maße to MASSE that the German language has an uppercase Eszett as well: ẞ This is as of now not widely deployed and few fonts support it, but in theory it is there now.

pdpi 2y ago

While we're having fun with that: # Capitalising an eszett changes the string length. >>> "straße".upper() 'STRASSE' # If you don't specify the locale, round-trip upper/lower case # messes up the dotless i used in turkic languages. >>> 'ı'.upper().lower() 'i'

kleiba 2y ago
Up until a few years ago, the first conversion would have been the official way to write a German word that contains the ligature ß (a lower-case letter) in all caps, because there was no corresponding upper-case letter. However, in 2017, an upper-case variant [1] was introduced into the official German orthography, so the conversion cited here should no longer be necessary. [1] https://en.wikipedia.org/wiki/Capital_%E1%BA%9E https://en.wikipedia.org/wiki/Capital_%E1%BA%9E
- Tomte 2y ago
  It is extremely unusual, and the addition to Unicode was very controversial. Basically, some type designers liked to play with capital ß and thought it cool to have it included into Unicode. There was a whole campaign for inclusion, and it was a big mistake. Because even though existing use must be shown to merit inclusion, they only managed to find a single book (a dictionary) printed in more than a few copies. From East Germany. And that book only used the capital ß for a single edition (out of dozens of editions) and reverted straight back to double s. Somehow, that still was enough for Unicode. Capital ß is a shit show, and it only confuses native speakers, because close to none of them have ever seen that glyph before. It has no real historic lineage (like the long s, for example, that pretty much nobody under 80 knows, either), it is a faux-retro modern design, an idle plaything.
  - kleiba 2y ago
    Well, what can you do... amtlich is amtlich. And the behavior of a function that converts ß to double upper-case S can certainly be discussed too, if only for the fact that a round-trip toUpper().toLower() will not give you back the original word.