4 ms·

I think the key to doing text sanely in programming is separating "text" from "international text" or "user text". "Text", can be e.g. the characters that make

I think the key to doing text sanely in programming is separating "text" from "international text" or "user text". "Text", can be e.g. the characters that make up my xml node names. Or all the names of my db columns etc. You still have to worry about encodings and everything with this data, but you don't have to worry that there is a 10 byte emoji or a turkish upper case i. A key property of it is: you can, for example, run toUpper or toLower, with a default culture. It has symmetric transforms. It can often be assumed to be the ASCII subset, regardless off encoding. Then on the other end you have text what the user enters. It can be anything (so may need validation and washing). You may not be able to run "to lower" on it (although I'd be tempted to do it on an email address for example). The key is just knowing what you have. It's unfortunate that "string" is usually used for everything from paths to user input to db column names etc.

HelloNurse 1y ago
> you don't have to worry that there is a 10 byte emoji or a turkish upper case i Excluding all emoji is silly but feasible (except for actual thorough custom validation and error handling of all inputs), but excluding some uppercase and lowercase letters because you don't feel up to the task of processing them is demeaning lunacy.
- alkonaut 1y ago
  Again, I'm now talking about "known" text in the programming context. Text that is neither user-input, or presented to a user. (E.g. column names. builtin function names in my toy spreadsheet. Whatever). Obviously for text that is both user-input and then displayed back again to users, you are in the other category. Apart from protecting against rendering mishaps and security etc, you probably just want to preserve what they write. But that was my point: 90% or more of the text you do is likely in the first category. And very rarely do you even have do deal with text in the second category.
pie_flavor 1y ago
If you want a separate language type for programmatic identifiers upon which all operations are defined in a culture-independent way, (a) good luck getting people to use it reliably, and (b) good luck with the numerous places where you have to convert back and forth and thus need all the information you're trying to not need. Your XML files start with `<?xml version="1.0" encoding="utf-8"?>`. If they then cannot actually support common UTF-8 sequences such as emoji or CJK characters, then your system is bugged, and you should fix it.
- alkonaut 1y ago
  The schema I meant was created in-house. I meant the tags and attribute names (schema) I can control. Just like the compiler controls what can go in a function name (but not in a string literal or comment).
eska 1y ago
That’s unfortunately what’s so wrong with Unicode, while I do appreciate a lot of its improvements. It’s not enough to have a global locale, but even a local per-string locale is not enough. The locale should be in-band, i.e. part of the text, so it can be switched like in a state machine

HelloNurse 1y ago

> you don't have to worry that there is a 10 byte emoji or a turkish upper case i Excluding all emoji is silly but feasible (except for actual thorough custom validation and error handling of all inputs), but excluding some uppercase and lowercase letters because you don't feel up to the task of processing them is demeaning lunacy.

alkonaut 1y ago
Again, I'm now talking about "known" text in the programming context. Text that is neither user-input, or presented to a user. (E.g. column names. builtin function names in my toy spreadsheet. Whatever). Obviously for text that is both user-input and then displayed back again to users, you are in the other category. Apart from protecting against rendering mishaps and security etc, you probably just want to preserve what they write. But that was my point: 90% or more of the text you do is likely in the first category. And very rarely do you even have do deal with text in the second category.

pie_flavor 1y ago

If you want a separate language type for programmatic identifiers upon which all operations are defined in a culture-independent way, (a) good luck getting people to use it reliably, and (b) good luck with the numerous places where you have to convert back and forth and thus need all the information you're trying to not need. Your XML files start with `<?xml version="1.0" encoding="utf-8"?>`. If they then cannot actually support common UTF-8 sequences such as emoji or CJK characters, then your system is bugged, and you should fix it.

alkonaut 1y ago
The schema I meant was created in-house. I meant the tags and attribute names (schema) I can control. Just like the compiler controls what can go in a function name (but not in a string literal or comment).

eska 1y ago

That’s unfortunately what’s so wrong with Unicode, while I do appreciate a lot of its improvements. It’s not enough to have a global locale, but even a local per-string locale is not enough. The locale should be in-band, i.e. part of the text, so it can be switched like in a state machine