14 ms·
An email falsehood surprised me recently: I thought a case-insensitive email address can be compared by using pseudocode `lower(x)`. But that's false. An email
by jph 4y ago
An email falsehood surprised me recently: I thought a case-insensitive email address can be compared by using pseudocode `lower(x)`. But that's false.
An email system that guarantees case-insensitive email addresses can still fail during comparisons of lowercase-to-lowercase, due to international encodings, locales, I18N, L10N, etc.
Pseudocode:
string-compare-case-insensitive(x, y) => true // Right way
string-compare(lower(x), lower(y)) => false // Wrong way
It turns out this issue is called a "case folding non-deterministic" error, and is a broader issue with strings in general.
For more about "case folding" with Unicode: http://www.unicode.org/Public/UNIDATA/CaseFolding.txt http://www.unicode.org/Public/UNIDATA/CaseFolding.txt
For more about "non-deterministic" comparisons: https://www.postgresql.org/docs/current/collation.html#COLLATION-NONDETERMINISTIC https://www.postgresql.org/docs/current/collation.html#COLLA...
- landofredwater 4y agoWhat would the mechanism behind case-insesitive string compare be? How would you program out all the edge cases?
- connicpu 4y agoPresumably the case-insensitive version is also doing unicode normalization as well, which is what a byte-level comparison of tolower versions would miss
- School-Cotton 4y agoBy implementing the Unicode’s case folding specification in detail, as described in section 5.18 of the Unicode standard. Or, more likely, by using a library like ICU.
- silon42 4y agoit is host specific.
- bmn__ 4y agoUnicode does not depend on a host. You probably meant something else, and the expression came out wrong. Care to explain in more detail?
- PeterisP 4y agoAs Unicode standard describes (e.g. the same 5.18 section mentioned above) case mapping depends on locale, so lowercasing the same string may have different results on different hosts, and so also the truthfulness of lowercase(x)==lowercase(y) is not universal and depends on the host locale. See the standard https://www.unicode.org/versions/Unicode11.0.0/ch05.pdf https://www.unicode.org/versions/Unicode11.0.0/ch05.pdf for the most commonly used example of Turkish i, but there are others.
- Quekid5 4y agoIndeed, a fundamental to the problem is that most unicode text doesn't actually carry the relevant locale information... (Of course, one probably wouldn't want to rely on sender-specified locales for email adresses when deciding address equality -- that would open one up to all sorts of potential weird scenarios, i.e. a nightmare for security).
- Dylan16807 4y agoIf you're setting up proper case folding, part of your job is not leaving locale up to the host.
- kubanczyk 4y agoA primer (taken straight from GP's first link): > the full case foldings are superior: for example, they allow "MASSE" and "Maße" to match.
- skupig 4y agoThis sounds like a fun vulnerability to find in a password reset flow
- thrdbndndn 4y agoYeah but isn't email address in ascii? I still have no idea why it would be different.
- tssva 4y agoAnother falsehood to add to the list.
- School-Cotton 4y ago> isn't email address in ascii? Yes in the basic SMTP RFC, but there are extensions that allow non-ascii local-parts.
- systemvoltage 4y agoThe whole thing is a mess. I remember trying to make a Postgres email address column and wanted to do make sure it can do comparisons either way, then found this stackoverflow post that shattered my expectations of a clean well understood problem: https://dba.stackexchange.com/questions/68266/what-is-the-best-way-to-store-an-email-address-in-postgresql https://dba.stackexchange.com/questions/68266/what-is-the-be...
- jph 4y agoYes you're right, same kind of issue. Thank you for the link with the detail.
- Gigachad 4y agoCase insensitivity was a huge mistake in computing really. Most languages don't have cases and its very non trivial to convert between cases. Should have treated every char as completely unique. Sure. At the user signup side, block emails that are too close in ways like case, but as a sender you should always treat them as unique emails.
- wereHamster 4y agotolower() works mostly well in ASCII and languages which don't have weird rules (=works mostly well in english).
- 3np 4y agoThe context is e-mail addresses, not English.
- stouset 4y agoI think that's GP's point: tolower() looks like it works well to English speakers but it's subtly wrong and will fail unexpectedly for people with other locales.
- gumby 4y agoThose rules are only weird to you, but perfectly reasonable to others, such as Turkish or German speakers…well, readers :-)
- msh 4y agoInternalization is hard. I think its too much to expect software written for a specific market to handle all languages in the world. Fx in danish we have 3 letters (æøå) that is not common in the latin alphabet. I cant go to germany or turkey and expect people to be able to write out those letters when doing input in a local system.
- MandieD 4y agoFun thing I've run into in a Germany-based but increasingly international company: German always spells out umlauts and eszetts when going to lower ASCII for email addresses ("Schäßler" -> "Schaessler"), but Hungarian does not. Not sure how Turkish ö and ü get fully lower-ASCII-ized there, but in Germany, they get spelled out "oe" and "ue" as if they were German ö and ü. This isn't as much of a corner case as one might think - there are a lot of people with Turkish names in Germany.
- heurisko 4y ago> string-compare-case-insensitive(x, y) => true // Right way what is inside that function?
- amenghra 4y agofunc string-compare-case-insensitive(x, y) { return lower(x) == lower(y) ? lower(x) == lower(y) : lower(x) == lower(y); } /s More seriously, if you want to learn about this, one place is to look at OpenJDK's sources. E.g. here: https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/String.java#L1978 https://github.com/openjdk/jdk/blob/master/src/java.base/sha... which calls: https://github.com/openjdk/jdk/blob/17283cfe4c697e2118f19992a6e87dbee268061e/src/java.base/share/classes/java/lang/String.java#L2141 https://github.com/openjdk/jdk/blob/17283cfe4c697e2118f19992... and https://github.com/openjdk/jdk/blob/17283cfe4c697e2118f19992a6e87dbee268061e/src/java.base/share/classes/java/lang/String.java#L2230 https://github.com/openjdk/jdk/blob/17283cfe4c697e2118f19992...
- jph 4y agoWhat's inside that function is Unicode case folding and non-deterministic collation, such as ICU, or its equivalent. See https://icu.unicode.org https://icu.unicode.org
- kuon 4y agoI think the part before the @ is actually case sensitive per RFC, but most mail server will treat it case insensitive. But I am not sure I am reading the RFC correctly, citation: Verbs and argument values (e.g., "TO:" or "to:" in the RCPT command and extension name keywords) are not case sensitive, with the sole exception in this specification of a mailbox local-part.
- a-dub 4y agorfc 822 local-parts (stuff to the left of @) are weirdly permissive. you can include quoted spaces, carriage returns and newlines (!)
- jwilk 4y agoQuoting CR and LF is no longer allowed in RFC 2822.
- jwilk 4y agoCorrection: it's still allowed, only considered obsolete. :-/
- derefr 4y agoWhich is to say, nobody of note is still allowing it; you can allow it if you like, for anyone who is still doing it; but if you don't, and someone sends you a message, and it bounces, then, well, they already know why.
- marcosdumay 4y agoYes, there are very few rules on the standard that could create two binary-different local parts that resolve to the same value (mostly involving parenthesis). But the mail server can add as many rules it wants, including the very common setting of making completely different names resolve to the same mailbox.
- tremon 4y agoI thought a case-insensitive email address can be compared by using pseudocode lower(x) You shouldn't be comparing the mailbox part of email addresses at all other than as literal bytestrings: you cannot know what equivalence rules the mailserver for that domain uses. The domain part can be equivalence-tested using the normal rules for domains though, including case insensitivity, IDN translation and punycode resolution.
- jph 4y ago> you cannot know what equivalence rules the mailserver for that domain uses. You're right in general. In my specific post, I do know the equivalence rules, because I'm administering the mail system and working with its source code, and the documentation guarantees/requires that internally its email addresses are treated entirely as case-insensitive. What I saw was source code comments about not using `lower(x)` nor Postgres module `citext`, and instead using Unicode case folding ICU and Postgres non-deterministic collations. In the end, what surprised me wasn't about email servers in general, it was about human languages with case folding.
- joshdata 4y ago> You shouldn't be comparing the mailbox part of email addresses at all other than as literal bytestrings It's hard to know what to do in practice, but this seems to be wrong according to https://www.rfc-editor.org/rfc/rfc6532#section-3.1 https://www.rfc-editor.org/rfc/rfc6532#section-3.1: "normalization form NFC SHOULD be used".
- tsimionescu 4y agoThis also depends a lot on why you are comparing those addresses. If you want for example to make sure that you don't easily allow the same person to register multiple accounts (say, to take advantage of a free trial period), then they are both wrong, since X@gmail.com, X.@gmail.com, X..@gmail.com etc. are all the same account and cost nothing to make. However, if you just want to make sure this is the same user that signed in earlier, you get to chose what rules you want - it's their problem to some extent to remember what account name they gave you.
- Asraelite 4y agoJust a nitpick: Gmail doesn't allow consecutive periods, so X..@gmail.com doesn't work. You can do a.bc, a.b.c, ab.c etc. instead.
- tsimionescu 4y agoOops, you're right. Did they also support user+somestring@gmail.com to be equal to user@gmail.com as well?
- Asraelite 4y agoYeah, they support +something. Sadly lot of websites don't accept it as a valid email address because of the +.
- imglorp 4y agoI wonder if that's because they know people use + to tag for later spam blocking, or is it just a lazy regex user who needs to read this article?
- skeeter2020 4y agoIME it's the latter, but not because they are lazy, but because email validation is hard. The lazy regex was in the late 90's early 00's when any email that didn't end in .com, .edu or .net failed!
- deleted 4y ago[deleted]