4 ms·
I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strin
by NovemberWhiskey 3y ago
I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strings were null-terminated byte streams. The platform had CSV import capability for tabular data, and it had an integrated pivot table capability based on some widgets that had been grafted onto it.
Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this.
My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.
- GuB-42 3y agoUTF-8 is made in such a way that processing based on ASCII and bytes mostly works. All non-ASCII characters are just a sequence of bytes >127, and the system can recover if code points are split or truncated. So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.