10 ms·

Creative usernames and Spotify account hijacking

281 points by e1ven 13y ago

johnvschmitt 13y ago
This seems odd. I mean, if their code was properly modular, they would have just one place where they "fetchUserIdByName(userName)", which returns one user ID or null if it's not used yet. When a new user is created, it then gets assigned a unique user ID. The email address is assigned to that user ID. Then, if they do a password reset on user = "bigbird", it should do the exact same lookup to find the email address. The security bug was not really about having an improper function to do unicode translation. It was more about having different functions for the same check, simply because they were in different parts of the code. Modular code is just so much better on all fronts, including security.
- eli 13y ago
  Easy to implement in a small project. Not so easy in a large enterprise project likely comprised of multiple interconnected systems.
  - TillE 13y ago
    Not a good excuse. There are plenty of enormous projects that provide only one basic interface to a bit of information. See every operating system API for examples.
    - jzwinck 13y ago
      How about the API to allocate memory in Windows? VirtualAlloc: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... VirtualAllocEx: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366890(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... VirtualAllocExNuma: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366891(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... It all started out nice and clean I'm sure, but within a few years you start to see many more than one basic interface to some things.
      - brokenparser 13y ago
        And {Global,Heap,Local}{,re}{Alloc,Free,Lock,Unlock}. 4 sets of APIs just to allocate memory. Although Local* and Global* are mapped to Heap for a while now and (IIRC) (un)lock functions don't do anything (but they used to). This BC is still required today.
- dan_manges 13y ago
  Based on their description of the bug, it sounded like the code was modular, but they called the function twice: once when the password reset request was generated, and again when the link in the email was clicked. However, when the link was used, canonical_username was once again applied So after they sent the password reset link, they called "fetchUserIdByName" again, but they passed in a username that had already been canonicalized once. Because of this bug, I wonder if password resets worked at all for users with unicode characters in their names.
  - mmahemoff 13y ago
    If you're saying canonicalise(canonicalise(name)) is not the same as canonicalise(name), that's going to be seriously bug-prone. Idempotence ftw.
    - deleted 13y ago
      [deleted]
    - spicyj 13y ago
      That's exactly what they describe as the cause of the bug. They intended for the function to be idempotent but it wasn't because of a misunderstanding with the Python library spec.
      - Xorlev 13y ago
        Worse, it /did/ work that way in Python 2.4 but Python 2.5 stopped throwing an exception for invalid codepoints which broke the Twisted library which broke their canonicalization function.
    - sanderjd 13y ago
      You should read the article. It prominently features a very interesting description of precisely why their `canonicalise` function turned out to not be idempotent, even though it was meant to be.
    - frogpelt 13y ago
- shawnz 13y ago
  This is what I thought at first. If it's the exact same check (which it should be), why is there any possibility of the answer being different? But the real "bug" here is that they were treating the canonicalized username as if it were just as good as the original username, which is only true if the adjustment function is idempotent as they say. Another possible solution would be to assume the username given to the password reset form is already canonicalized (which would be necessarily true, as far as I understand). EDIT: However this wouldn't solve the other bug that's been discovered here, which is that "ᴮᴵᴳᴮᴵᴿᴰ" is canonicalized differently than "BIGBIRD", thus defeating the purpose of canonicalization (for that particular case) in the first place.
- brown9-2 13y ago
  Sounds like they are using the username as the key in their DBs, which sounds like the ultimate case of any pain: Could the method for computing canonical usernames based on nodeprep.prepare() be salvaged? If not we would be in trouble since we use canonical usernames in various databases so that changing how to derive them in a non-backwards compatible way would be quite costly.
  - tantalor 13y ago
    Not necessarily. It their canonicalization function were idempotent (e.g., the identity), then this database scheme would work well. How else do you map username to user id?
    - brown9-2 13y ago
      Well generally it makes sense to use that userid as a primary key everywhere rather than the username. You only need the mapping of username to userid in one place. Their current scheme also makes it sound hard to change your username.
      - tantalor 13y ago
        Ah I fully agree. I misunderstood your post to say you should never use a user name as a key.
- tantalor 13y ago
  What you're talking about has nothing to do with modularity. You're thinking of DRY. They having nothing to do with each other. DRY tells us the code for looking up a user id by user name should be written in only one place, not that the code should be called from only one place. The mistake was assuming the name->name function was idempotent, because it wasn't. You are right to suggest using a name->id function instead. It would not suffer the same problem because the canonical name should not be stored... it's an implementation detail!
  - Confusion 13y ago
    They having nothing to do with each other. DRY tells us the code for looking up a user id by user name should be written in only one place, not that the code should be called from only one place. If you write 'canonicalize(username)' in eight different places, you are not being DRY. If you need to write 'canonicalize(username)' in eight different places, your code probably doesn't separate responsibilities properly across separate modules. As such, they have a lot to do with each other. After all, if you call code from more than one place, you are writing the calling code in more than one place. Lack of DRYness is about the fact that you are doing so. Lack af modularity is about why you need to do so.
Jhsto 13y ago
I'd like to point out that even though not every website accepts unicode characters on registration, some software such as vBulletin let the administrator/user to replace the username with such characters once the user is signed up. You may not create as much havoc as in the original post, but some level of confusion at least.
pilsetnieks 13y ago
I'm confused about why they even had the need for canonicalising the usernames for whatever purpose? Why couldn't they just have stored and used them just as they are?
- dan_manges 13y ago
  They gave one reason in the post. They wanted usernames to be case insensitive, so that if there's a user named BigBird, somebody else can't sign up as bigbird. Case insensitive usernames are also helpful to minimize support issues when somebody forgets the exact case they used when they created their account.
  - drivebyacct2 13y ago
    Am I insensitive in wondering "Why not just limit it to lowercase, ASCII chars"?
    - EvanKelly 13y ago
      I don't think you're insensitive, but on a site which wants to foster users who care about their profile and persona on the site, freedom to display a username how they like could be a valuable feature.
    - mzl 13y ago
      Yes, because lowercase ASCII does not meaningfully capture how names are written in most of the world. If you want to cater to a global audience, it is simply not sufficient.
    - willvarfar 13y ago
      As they say in the article, they are trying to serve a global audience. As is easily googlable, Spotify was developed in the Baltic; the developer's own names likely contain non-ASCII characters.
      - drivebyacct2 13y ago
United857 13y ago
A similar issue exists with international domain names. https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack
- Groxx 13y ago
  it's a different problem, though to some degree they stem from the same underlying issue (similar but different characters).
Esifer 13y ago
Why does Unicode threat Omega and Ohm like different characters?
- eli 13y ago
 Why does HTML have a and a tag?
 - sleepyhead 13y ago
 is emphasis/semantic and is visual/style.
 - oneeyedpigeon 13y ago
 The parent was asking a rhetorical question, and drawing an analogy between the two: omega and ohm have different meanings, therefore it might be useful to be able to distinguish between the two. (Note that I say 'might'; when it comes to a character set, I'm not sure I 100% agree ...)
- drrotmos 13y ago
 Because one is a greek letter, and the other is a unit of resistance. The fact that they're represented by the same symbol is irrelevant.
 - threedaymonk 13y ago
 If that were the case, we'd need separate codepoints for every letter used as a unit of measurement, from A for ampere onwards. In fact, it's just there for legacy reasons, as pointed out elsewhere: the convention for units of measurement is to use normal letters, regardless of whether those are Latin or Greek.
- duskwuff 13y ago
 Because they have rather different meanings. Capital omega (Ω, U+03A9) is a Greek letter, with the lower-case form ω; ohm (Ω, U+2126) is a symbol used in electrical engineering with a related symbol "mho" (℧, U+2127).
deleted 13y ago
[deleted]
gkoberger 13y ago
Reminds me of the early GMail bug, where you could sign up with "john.smith", "jo.hn.smi.th" or "johnsmith", but all mail was delivered to the period-less version: http://arstechnica.com/uncategorized/2006/01/6022-2/ http://arstechnica.com/uncategorized/2006/01/6022-2/
- knome 13y ago
  That isn't a bug. Even the article you link notes that it is a feature, though they seem for some reason incredulous. I've always liked that they do this on my accounts.
- Maxious 13y ago
  The link agrees - it's a feature not a bug and one that considered security: > As it turns out, even though Gmail will act as if there is no period in a username when delivering mail, it will not permit users to register accounts whose only difference to other account is a period. That is, if bob.jones is registered, bobjones cannot get an account (he could get bobjones2, obviously).
- praseodym 13y ago
  You could (and still can) only register one of those addresses. Mail from the other two will be delivered to that address. So, it's not a bug but rather a pretty nice feature: ever seen a handwritten email address with a period squeezed in there?
a-nikolaev 13y ago
I don't see any real reason to rely on idempotence. They could simply store two names: One is provided by the user (verbatim), and the second is its reduction to lowercase letters and digits (canonical). For all internal logic, they could use only the canonical name, and use the verbatim name in the front-end to make the user happy. > Lower casing has the key property of being idempotent, i.e., that applying it more than once has no effect: x.lower() == x.lower().lower(). So if a username gets passed from service to service and you want to make sure it is in canonical form you can safely apply .lower() and if it was already in canonical form there is no harm done, and it is easy to stay safe. Apparently, they thought that it's ok to use verbatim and canonical names interchangeably, relying on idenpotence property of the XMPP function. It looks like a huge design error.
- readme 13y ago
  >They could simply store two names Presumably you mean in the database? I don't see a reason to keep a copy of the lower() transformation of a string when it is incredibly cheap to transform a small string to lowercase. What exactly is the point of that? I would just call lower() as needed, personally.
  - a-nikolaev 13y ago
    Oh, my point was that two kinds of names should not be used interchangeably. To store or not the .lower() is a matter of taste.. (Personally, I would store both names, just to avoid wasting computing time)
    - readme 13y ago
      I think the act of storing both names is bad, because you multiply the amount of data that could possibly become wrong by 2. With lower(), we can expect we'll get the right transformation of string A each time. If instead, we store string A, and then store string B as A.lower() and copy it... A.lower() will always be A.lower, but it's much easier for someone to come along, screw with the database, and change B.
drivebyacct2 13y ago
So is there a need for a TRULY idempotent equivalent of XMPP's nodeprep? Or one that handles more Unicode points? Or is it a calculated decision to support Unicode 3.2 points only? (Sorry for the nooby questions, but this was very interesting and I don't know a lot about Unicode)
- acjohnson55 13y ago
  Correct. And it would seem to me that such a function should probably be in its own separate library, or perhaps the standard library. It seems like a fairly basic need.
- lancestout 13y ago
  The decision to only support Unicode 3.2 is simply because the StringPrep framework [1] (which XMPP's nodeprep and various other protocols use) is forever tied to that version of Unicode. Current work is on the PRECIS framework [2] which uses the metadata for Unicode code points to determine how to handle them during canonicalization instead of relying on a hard coded set of mapping tables. There's still a lot of work to be done, mainly to review that the process works reliably and doesn't introduce subtle new issues. Peter Saint-Andre (one of the authors of PRECIS) has just started on a Python tool for testing how a given version of Unicode is handled by PRECIS (https://github.com/stpeter/PrecisMaker https://github.com/stpeter/PrecisMaker). [1] https://www.ietf.org/rfc/rfc3454.txt https://www.ietf.org/rfc/rfc3454.txt [2] https://tools.ietf.org/html/draft-ietf-precis-framework-08 https://tools.ietf.org/html/draft-ietf-precis-framework-08
  - drivebyacct2 13y ago
    Great information, even cooler to see someone's got some code working alongside it, I might have to adapt it to Go if it's fairly reasonable to understand given my relative lack of experience with Unicode (heh as I'd mentioned and is probably obvious given your knowledge on hand).
deleted 13y ago
[deleted]
waster 13y ago
A fascinating account.
tantalor 13y ago
So if a username gets passed from service to service and you want to make sure it is in canonical form you can safely apply .lower() and if it was already in canonical form there is no harm done, and it is easy to stay safe. Why? Suppose you only pass the original name around instead. Then you don't require your canonicalization function to be idempotent, which might be good in your case since it wasn't.
- antocv 13y ago
  Because they wanted "BigBird" to be the same as "bigbird" or "BIGBIRD". Hence if you wanted to rely on a userId, you would still need to ensure all the variations of BiGbirD map to the same userId.
NKCSS 13y ago
I think they could have handled the 'reward' a bit better :P "In this case the two users who posted to the forum where actually rewarded with some Spotify premium months." I'd say: Premium lifetime memberships would be better :)
- robmcm 13y ago
  Yeah, seeing as before they had the option to log into anyones account, affectivly the same as a lifetime membership.
Sami_Lehtinen 13y ago
Nothing new at all, even my stockbroker had similar kind of bug. There should be only one function which handles usernames. When same thing is implemented differently all over the source, this is exactly what happens. I have seen also much worse solutions. Where actually giving username, logs you in (sets logged in session cookie) and then prompts for password. When you enter invalid password you're logged out. If you give username, and then change url, you're in. Business as usual. When you test it, it works. Username + right password = ok, Username + wrong password != ok. Tests passed, and that's it.
paul_f 13y ago
We require an email address as a user name. Does this get around the problem?
- troels 13y ago
  Depends on how you validate an email address.
jrochkind1 13y ago
Very interesting post -- dealing with unicode gets tricky. Because dealing with a full character repertoire is tricky, not because unicode does it poorly (unicode actually does it well). I wonder if Unicode UTR#30 would be an alternative (more reliable?) method of idempotent normalization? http://www.unicode.org/reports/tr30/tr30-4.html http://www.unicode.org/reports/tr30/tr30-4.html Or maybe the twisted algorithm really is UTR#30, just not labelled that? As far as I can tell, UTR#30 did not make it to formally being part of the unicode spec, for reasons I'm not entirely clear on -- it is nonetheless quite useful, and this case is an example. Solr for instance still uses it. (http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ICUFoldingFilterFactory http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#...). It might be a pain to find code implementing UTR#30 in your language of choice though (I am not sure if it's part of current ICU libraries or not). It's also worth pointing out, that in addition to this kind of 'folding' of different-but-look-the-same graphemes, in this sort of use case you ABSOLUTELY need to do byte normalization as per UAX#15 http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ . Probably NFKC for this sort of use case.
- jrochkind1 13y ago
  Huh, and another alternative would be using one of the unicode collation algorithms for normalization -- which, unlike UAX#15, did make it beyond draft stage to be an official part of the unicode spec. http://www.unicode.org/reports/tr30/tr30-4.html#_Toc23 http://www.unicode.org/reports/tr30/tr30-4.html#_Toc23
nodata 13y ago
> it is probably good to... treat “BigBird” and “bigbird” as the same username. Erm...
- Groxx 13y ago
  How about "building" and "buiIding"? or "nodata" and "ｎodata"? You probably don't want people running around claiming to be admins because their name looks the same. Similarly, do you really want to do tech support when someone forgets that their username is "nodata" instead of "Nodata", since their phone auto-capitalized things when they signed up? It happens. And how do you know they're "Nodata" and not "nOdata" or "nOdAtA"?
dhtbccnhmdhm 13y ago
Why not to make canonization while the names differ? canName=canonical_username(name); for(canName1=canName,i=0;canName1!=canName;i++){ canName1=canonical_username(canName); if(i>=treshold){ stop_registration(); break; } }

281 points by e1ven 13y ago

This seems odd. I mean, if their code was properly modular, they would have just one place where they "fetchUserIdByName(userName)", which returns one user ID or null if it's not used yet. When a new user is created, it then gets assigned a unique user ID. The email address is assigned to that user ID. Then, if they do a password reset on user = "bigbird", it should do the exact same lookup to find the email address. The security bug was not really about having an improper function to do unicode translation. It was more about having different functions for the same check, simply because they were in different parts of the code. Modular code is just so much better on all fronts, including security.

eli 13y ago
Easy to implement in a small project. Not so easy in a large enterprise project likely comprised of multiple interconnected systems.
- TillE 13y ago
  Not a good excuse. There are plenty of enormous projects that provide only one basic interface to a bit of information. See every operating system API for examples.
  - jzwinck 13y ago
    How about the API to allocate memory in Windows? VirtualAlloc: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... VirtualAllocEx: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366890(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... VirtualAllocExNuma: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366891(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... It all started out nice and clean I'm sure, but within a few years you start to see many more than one basic interface to some things.
    - brokenparser 13y ago
      And {Global,Heap,Local}{,re}{Alloc,Free,Lock,Unlock}. 4 sets of APIs just to allocate memory. Although Local* and Global* are mapped to Heap for a while now and (IIRC) (un)lock functions don't do anything (but they used to). This BC is still required today.
dan_manges 13y ago
Based on their description of the bug, it sounded like the code was modular, but they called the function twice: once when the password reset request was generated, and again when the link in the email was clicked. However, when the link was used, canonical_username was once again applied So after they sent the password reset link, they called "fetchUserIdByName" again, but they passed in a username that had already been canonicalized once. Because of this bug, I wonder if password resets worked at all for users with unicode characters in their names.
- mmahemoff 13y ago
  If you're saying canonicalise(canonicalise(name)) is not the same as canonicalise(name), that's going to be seriously bug-prone. Idempotence ftw.
  - deleted 13y ago
    [deleted]
  - spicyj 13y ago
    That's exactly what they describe as the cause of the bug. They intended for the function to be idempotent but it wasn't because of a misunderstanding with the Python library spec.
    - Xorlev 13y ago
      Worse, it /did/ work that way in Python 2.4 but Python 2.5 stopped throwing an exception for invalid codepoints which broke the Twisted library which broke their canonicalization function.
  - sanderjd 13y ago
    You should read the article. It prominently features a very interesting description of precisely why their `canonicalise` function turned out to not be idempotent, even though it was meant to be.
  - frogpelt 13y ago
shawnz 13y ago
This is what I thought at first. If it's the exact same check (which it should be), why is there any possibility of the answer being different? But the real "bug" here is that they were treating the canonicalized username as if it were just as good as the original username, which is only true if the adjustment function is idempotent as they say. Another possible solution would be to assume the username given to the password reset form is already canonicalized (which would be necessarily true, as far as I understand). EDIT: However this wouldn't solve the other bug that's been discovered here, which is that "ᴮᴵᴳᴮᴵᴿᴰ" is canonicalized differently than "BIGBIRD", thus defeating the purpose of canonicalization (for that particular case) in the first place.
brown9-2 13y ago
Sounds like they are using the username as the key in their DBs, which sounds like the ultimate case of any pain: Could the method for computing canonical usernames based on nodeprep.prepare() be salvaged? If not we would be in trouble since we use canonical usernames in various databases so that changing how to derive them in a non-backwards compatible way would be quite costly.
- tantalor 13y ago
  Not necessarily. It their canonicalization function were idempotent (e.g., the identity), then this database scheme would work well. How else do you map username to user id?
  - brown9-2 13y ago
    Well generally it makes sense to use that userid as a primary key everywhere rather than the username. You only need the mapping of username to userid in one place. Their current scheme also makes it sound hard to change your username.
    - tantalor 13y ago
      Ah I fully agree. I misunderstood your post to say you should never use a user name as a key.
tantalor 13y ago
What you're talking about has nothing to do with modularity. You're thinking of DRY. They having nothing to do with each other. DRY tells us the code for looking up a user id by user name should be written in only one place, not that the code should be called from only one place. The mistake was assuming the name->name function was idempotent, because it wasn't. You are right to suggest using a name->id function instead. It would not suffer the same problem because the canonical name should not be stored... it's an implementation detail!
- Confusion 13y ago
  They having nothing to do with each other. DRY tells us the code for looking up a user id by user name should be written in only one place, not that the code should be called from only one place. If you write 'canonicalize(username)' in eight different places, you are not being DRY. If you need to write 'canonicalize(username)' in eight different places, your code probably doesn't separate responsibilities properly across separate modules. As such, they have a lot to do with each other. After all, if you call code from more than one place, you are writing the calling code in more than one place. Lack of DRYness is about the fact that you are doing so. Lack af modularity is about why you need to do so.

Jhsto 13y ago

I'd like to point out that even though not every website accepts unicode characters on registration, some software such as vBulletin let the administrator/user to replace the username with such characters once the user is signed up. You may not create as much havoc as in the original post, but some level of confusion at least.

pilsetnieks 13y ago

I'm confused about why they even had the need for canonicalising the usernames for whatever purpose? Why couldn't they just have stored and used them just as they are?

dan_manges 13y ago
They gave one reason in the post. They wanted usernames to be case insensitive, so that if there's a user named BigBird, somebody else can't sign up as bigbird. Case insensitive usernames are also helpful to minimize support issues when somebody forgets the exact case they used when they created their account.
- drivebyacct2 13y ago
  Am I insensitive in wondering "Why not just limit it to lowercase, ASCII chars"?
  - EvanKelly 13y ago
    I don't think you're insensitive, but on a site which wants to foster users who care about their profile and persona on the site, freedom to display a username how they like could be a valuable feature.
  - mzl 13y ago
    Yes, because lowercase ASCII does not meaningfully capture how names are written in most of the world. If you want to cater to a global audience, it is simply not sufficient.
  - willvarfar 13y ago
    As they say in the article, they are trying to serve a global audience. As is easily googlable, Spotify was developed in the Baltic; the developer's own names likely contain non-ASCII characters.
    - drivebyacct2 13y ago

United857 13y ago

A similar issue exists with international domain names. https://en.wikipedia.org/wiki/IDN_homograph_attack https://en.wikipedia.org/wiki/IDN_homograph_attack

Groxx 13y ago
it's a different problem, though to some degree they stem from the same underlying issue (similar but different characters).

Esifer 13y ago

Why does Unicode threat Omega and Ohm like different characters?

eli 13y ago
Why does HTML have a and a tag?
- sleepyhead 13y ago
 is emphasis/semantic and is visual/style.
 - oneeyedpigeon 13y ago
 The parent was asking a rhetorical question, and drawing an analogy between the two: omega and ohm have different meanings, therefore it might be useful to be able to distinguish between the two. (Note that I say 'might'; when it comes to a character set, I'm not sure I 100% agree ...)
drrotmos 13y ago
Because one is a greek letter, and the other is a unit of resistance. The fact that they're represented by the same symbol is irrelevant.
- threedaymonk 13y ago
  If that were the case, we'd need separate codepoints for every letter used as a unit of measurement, from A for ampere onwards. In fact, it's just there for legacy reasons, as pointed out elsewhere: the convention for units of measurement is to use normal letters, regardless of whether those are Latin or Greek.
duskwuff 13y ago
Because they have rather different meanings. Capital omega (Ω, U+03A9) is a Greek letter, with the lower-case form ω; ohm (Ω, U+2126) is a symbol used in electrical engineering with a related symbol "mho" (℧, U+2127).

deleted 13y ago

[deleted]

gkoberger 13y ago

Reminds me of the early GMail bug, where you could sign up with "john.smith", "jo.hn.smi.th" or "johnsmith", but all mail was delivered to the period-less version: http://arstechnica.com/uncategorized/2006/01/6022-2/ http://arstechnica.com/uncategorized/2006/01/6022-2/

knome 13y ago
That isn't a bug. Even the article you link notes that it is a feature, though they seem for some reason incredulous. I've always liked that they do this on my accounts.
Maxious 13y ago
The link agrees - it's a feature not a bug and one that considered security: > As it turns out, even though Gmail will act as if there is no period in a username when delivering mail, it will not permit users to register accounts whose only difference to other account is a period. That is, if bob.jones is registered, bobjones cannot get an account (he could get bobjones2, obviously).
praseodym 13y ago
You could (and still can) only register one of those addresses. Mail from the other two will be delivered to that address. So, it's not a bug but rather a pretty nice feature: ever seen a handwritten email address with a period squeezed in there?

a-nikolaev 13y ago

I don't see any real reason to rely on idempotence. They could simply store two names: One is provided by the user (verbatim), and the second is its reduction to lowercase letters and digits (canonical). For all internal logic, they could use only the canonical name, and use the verbatim name in the front-end to make the user happy. > Lower casing has the key property of being idempotent, i.e., that applying it more than once has no effect: x.lower() == x.lower().lower(). So if a username gets passed from service to service and you want to make sure it is in canonical form you can safely apply .lower() and if it was already in canonical form there is no harm done, and it is easy to stay safe. Apparently, they thought that it's ok to use verbatim and canonical names interchangeably, relying on idenpotence property of the XMPP function. It looks like a huge design error.

readme 13y ago
>They could simply store two names Presumably you mean in the database? I don't see a reason to keep a copy of the lower() transformation of a string when it is incredibly cheap to transform a small string to lowercase. What exactly is the point of that? I would just call lower() as needed, personally.
- a-nikolaev 13y ago
  Oh, my point was that two kinds of names should not be used interchangeably. To store or not the .lower() is a matter of taste.. (Personally, I would store both names, just to avoid wasting computing time)
  - readme 13y ago
    I think the act of storing both names is bad, because you multiply the amount of data that could possibly become wrong by 2. With lower(), we can expect we'll get the right transformation of string A each time. If instead, we store string A, and then store string B as A.lower() and copy it... A.lower() will always be A.lower, but it's much easier for someone to come along, screw with the database, and change B.

drivebyacct2 13y ago

So is there a need for a TRULY idempotent equivalent of XMPP's nodeprep? Or one that handles more Unicode points? Or is it a calculated decision to support Unicode 3.2 points only? (Sorry for the nooby questions, but this was very interesting and I don't know a lot about Unicode)

acjohnson55 13y ago
Correct. And it would seem to me that such a function should probably be in its own separate library, or perhaps the standard library. It seems like a fairly basic need.
lancestout 13y ago
The decision to only support Unicode 3.2 is simply because the StringPrep framework [1] (which XMPP's nodeprep and various other protocols use) is forever tied to that version of Unicode. Current work is on the PRECIS framework [2] which uses the metadata for Unicode code points to determine how to handle them during canonicalization instead of relying on a hard coded set of mapping tables. There's still a lot of work to be done, mainly to review that the process works reliably and doesn't introduce subtle new issues. Peter Saint-Andre (one of the authors of PRECIS) has just started on a Python tool for testing how a given version of Unicode is handled by PRECIS (https://github.com/stpeter/PrecisMaker https://github.com/stpeter/PrecisMaker). [1] https://www.ietf.org/rfc/rfc3454.txt https://www.ietf.org/rfc/rfc3454.txt [2] https://tools.ietf.org/html/draft-ietf-precis-framework-08 https://tools.ietf.org/html/draft-ietf-precis-framework-08
- drivebyacct2 13y ago
  Great information, even cooler to see someone's got some code working alongside it, I might have to adapt it to Go if it's fairly reasonable to understand given my relative lack of experience with Unicode (heh as I'd mentioned and is probably obvious given your knowledge on hand).

deleted 13y ago

[deleted]

waster 13y ago

A fascinating account.

tantalor 13y ago

So if a username gets passed from service to service and you want to make sure it is in canonical form you can safely apply .lower() and if it was already in canonical form there is no harm done, and it is easy to stay safe. Why? Suppose you only pass the original name around instead. Then you don't require your canonicalization function to be idempotent, which might be good in your case since it wasn't.

antocv 13y ago
Because they wanted "BigBird" to be the same as "bigbird" or "BIGBIRD". Hence if you wanted to rely on a userId, you would still need to ensure all the variations of BiGbirD map to the same userId.

NKCSS 13y ago

I think they could have handled the 'reward' a bit better :P "In this case the two users who posted to the forum where actually rewarded with some Spotify premium months." I'd say: Premium lifetime memberships would be better :)

robmcm 13y ago
Yeah, seeing as before they had the option to log into anyones account, affectivly the same as a lifetime membership.

Sami_Lehtinen 13y ago

Nothing new at all, even my stockbroker had similar kind of bug. There should be only one function which handles usernames. When same thing is implemented differently all over the source, this is exactly what happens. I have seen also much worse solutions. Where actually giving username, logs you in (sets logged in session cookie) and then prompts for password. When you enter invalid password you're logged out. If you give username, and then change url, you're in. Business as usual. When you test it, it works. Username + right password = ok, Username + wrong password != ok. Tests passed, and that's it.

paul_f 13y ago

We require an email address as a user name. Does this get around the problem?

troels 13y ago
Depends on how you validate an email address.

jrochkind1 13y ago

Very interesting post -- dealing with unicode gets tricky. Because dealing with a full character repertoire is tricky, not because unicode does it poorly (unicode actually does it well). I wonder if Unicode UTR#30 would be an alternative (more reliable?) method of idempotent normalization? http://www.unicode.org/reports/tr30/tr30-4.html http://www.unicode.org/reports/tr30/tr30-4.html Or maybe the twisted algorithm really is UTR#30, just not labelled that? As far as I can tell, UTR#30 did not make it to formally being part of the unicode spec, for reasons I'm not entirely clear on -- it is nonetheless quite useful, and this case is an example. Solr for instance still uses it. (http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ICUFoldingFilterFactory http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#...). It might be a pain to find code implementing UTR#30 in your language of choice though (I am not sure if it's part of current ICU libraries or not). It's also worth pointing out, that in addition to this kind of 'folding' of different-but-look-the-same graphemes, in this sort of use case you ABSOLUTELY need to do byte normalization as per UAX#15 http://unicode.org/reports/tr15/ http://unicode.org/reports/tr15/ . Probably NFKC for this sort of use case.

jrochkind1 13y ago
Huh, and another alternative would be using one of the unicode collation algorithms for normalization -- which, unlike UAX#15, did make it beyond draft stage to be an official part of the unicode spec. http://www.unicode.org/reports/tr30/tr30-4.html#_Toc23 http://www.unicode.org/reports/tr30/tr30-4.html#_Toc23

nodata 13y ago

> it is probably good to... treat “BigBird” and “bigbird” as the same username. Erm...

Groxx 13y ago
How about "building" and "buiIding"? or "nodata" and "ｎodata"? You probably don't want people running around claiming to be admins because their name looks the same. Similarly, do you really want to do tech support when someone forgets that their username is "nodata" instead of "Nodata", since their phone auto-capitalized things when they signed up? It happens. And how do you know they're "Nodata" and not "nOdata" or "nOdAtA"?

dhtbccnhmdhm 13y ago

Why not to make canonization while the names differ? canName=canonical_username(name); for(canName1=canName,i=0;canName1!=canName;i++){ canName1=canonical_username(canName); if(i>=treshold){ stop_registration(); break; } }