3 ms·

To me that sounds more like canonicalization than sanitation. Depending on your requirements it might be fine to convert the input to a canonical form before pr

To me that sounds more like canonicalization than sanitation. Depending on your requirements it might be fine to convert the input to a canonical form before processing. If you do this, be certain to do it before validation so that you don't accidentally "canonicalize" validated input into something which wouldn't pass the validation checks. A key aspect of canonicalization compared to sanitation is that the result should be something that the user would consider equivalent to their original input. The most common offender in my experience is the abuse of case normalization, especially for data like email addresses which are not defined as case-insensitive (at least for the mailbox name) even if many servers treat them that way. If you don't preserve the original case (and other parts such as "+" labels whose meaning is defined by the mail server) the address may not work at all, or may result in sending messages to the wrong user. Names, as an intimate part of the user's identity, are another area where case normalization can sometimes prove annoying or even offensive. If some legacy system requires names to be entered as all-caps US-ASCII characters, fine, but at least don't turn "O'Conner" or "MacDouglas" into "O'conner" or "Macdouglas" in some misguided attempt to ensure that just the first letter is capitalized. (And in some situations the first letter shouldn't be capitalized, e.g. the "dos Santos" in "Giovani dos Santos Ramírez"[0]—which is a single surname, not two names.) [0] https://en.wikipedia.org/wiki/Giovani_dos_Santos https://en.wikipedia.org/wiki/Giovani_dos_Santos

talideon 5y ago
Oh, believe me. As somebody with a name that includes accents, and a surname that contains two words, with relatives whose names include internal capitalisation and apostrophes, I know _all_ about that. The thing is that canonicalisation is a kind of sanitisation. As you mentioned, I personally prefer it to be done in real time. Sometimes it can't, however, you have to resort to munging, which is on the nastier end of sanitisation. Here's a short story: AFNIC run the .fr registry, and they, unlike other registries, expect you to provide a contact's given name and surname separately. The joys of French bureaucracy. At my previous job (hosting provider and domain registrar), I built the company's domain management system. The systems in front of that didn't care about the form a person's name took so long as it was present, and most other domain registries are the same. There was no sensible way to get the applicant to enter them previously (this data was taken from the billing system). This necessitated that I build a library that could parse people's names, and I ended up developing a rather large number of heuristics for doing so as accurately as possible. It only covered the Latin alphabet, as that's all AFNIC would accept at the time, but it worked. The problem is that most don't put that kind of thought into data sanitisation, and do things such as those you mentioned. And that's why we can't have nice things.

talideon 5y ago

Oh, believe me. As somebody with a name that includes accents, and a surname that contains two words, with relatives whose names include internal capitalisation and apostrophes, I know _all_ about that. The thing is that canonicalisation is a kind of sanitisation. As you mentioned, I personally prefer it to be done in real time. Sometimes it can't, however, you have to resort to munging, which is on the nastier end of sanitisation. Here's a short story: AFNIC run the .fr registry, and they, unlike other registries, expect you to provide a contact's given name and surname separately. The joys of French bureaucracy. At my previous job (hosting provider and domain registrar), I built the company's domain management system. The systems in front of that didn't care about the form a person's name took so long as it was present, and most other domain registries are the same. There was no sensible way to get the applicant to enter them previously (this data was taken from the billing system). This necessitated that I build a library that could parse people's names, and I ended up developing a rather large number of heuristics for doing so as accurately as possible. It only covered the Latin alphabet, as that's all AFNIC would accept at the time, but it worked. The problem is that most don't put that kind of thought into data sanitisation, and do things such as those you mentioned. And that's why we can't have nice things.