3 ms·

> this design makes the common case much simpler at the expense of making the rare case harder In the age of emoji (and uhhhh, everyone who doesn't use English

> this design makes the common case much simpler at the expense of making the rare case harder In the age of emoji (and uhhhh, everyone who doesn't use English as their main language), I don't think your "rare case" is really that rare.

o11c 3y ago
Do note that emoji are a case where the "codepoint" approach fails catastrophically.
- gautamcgoel 3y ago
  Can you please elaborate?
Exoristos 3y ago
There must be statistics around how much of the data in the world is or could be Latin-1. I'm going to guess it's a very high percentage.
- pests 3y ago
  wat? Counting webpages (not bytes) the number is near 50% are English only and 50% are other languages. Only 20% of people worldwide speak English. I know other languages can fit inside ASCII but, really?
xwolfi 3y ago
On paper you're not wrong, but String used for localized text are a special subclass you can deal with separately. Most Strings that will cause you problems are, you know, technical: logs, name of subsystems, client ids, client-sourced API-provided values which change format across client etc. Those, in my experience, are always ASCII even in China, exactly because nobody wants to deal with too much crap. Display Strings are simpler to manipulate in most cases: load String from file or form, store back verbatim in DB or memory, you barely do anything other than straight copying, right ? The way I do in Java is that I always assume and enforce my strings to be ASCII single byte, and if I want to display something localized, somehow, it never really goes through any complex logic where I need to know the encoding: I copy the content with an encoding metadata, and the other side just displays it.
avgcorrection 3y ago
> In the age of emoji (and uhhhh, everyone who doesn't use English as their main language) Open a random NYT article of more than a hundred words. It won’t be in ASCII.

o11c 3y ago

Do note that emoji are a case where the "codepoint" approach fails catastrophically.

gautamcgoel 3y ago
Can you please elaborate?

Exoristos 3y ago

There must be statistics around how much of the data in the world is or could be Latin-1. I'm going to guess it's a very high percentage.

pests 3y ago
wat? Counting webpages (not bytes) the number is near 50% are English only and 50% are other languages. Only 20% of people worldwide speak English. I know other languages can fit inside ASCII but, really?

xwolfi 3y ago

On paper you're not wrong, but String used for localized text are a special subclass you can deal with separately. Most Strings that will cause you problems are, you know, technical: logs, name of subsystems, client ids, client-sourced API-provided values which change format across client etc. Those, in my experience, are always ASCII even in China, exactly because nobody wants to deal with too much crap. Display Strings are simpler to manipulate in most cases: load String from file or form, store back verbatim in DB or memory, you barely do anything other than straight copying, right ? The way I do in Java is that I always assume and enforce my strings to be ASCII single byte, and if I want to display something localized, somehow, it never really goes through any complex logic where I need to know the encoding: I copy the content with an encoding metadata, and the other side just displays it.

avgcorrection 3y ago

> In the age of emoji (and uhhhh, everyone who doesn't use English as their main language) Open a random NYT article of more than a hundred words. It won’t be in ASCII.