3 ms·
At my job I have to deal with an old system that invented its own encoding, named TSS. The idea was to unify multiple charsets and encodings into one, before Un
by danhau 1y ago
At my job I have to deal with an old system that invented its own encoding, named TSS. The idea was to unify multiple charsets and encodings into one, before Unicode was a thing. But instead of coming up with one big a charset and assigning codepoints plus an encoding scheme, they thought it was wise to just repackage other encodings and charsets. Think Matroska, but for text. And yes, I do mean charsets AND encodings. Sometimes they repackage an encoding, sometimes just a charset where the codepoints are the encoding.
TSS supports the ISO-8859 charsets and corresponding (but deviating) Windows codepages, traditional and simplified Chinese, half- and fullwidth Japanese, Korean via Wansung and Johab, and others I'm forgetting right now. And in newer version of the software, they also support Unicode, but using a custom encoding.
Thankfully a good chunk of all that is well documented, like the byte values introducing a fullwidth Japanese character, for example. But they don't describe what charset or encoding is actually used. EUC-JP? Shift-JIS? Turns out it's JIS X 0208. You'd think they would just use Shift-JIS, which gives them both full- and halfwidth Japanese in one shot, but no. They package those explicitly as JIS X 0208 and JIS X 0201. Similar questions arise for Chinese and the others. It took a lot of reverse engineering to figure that stuff out. But if you think that is hard, have fun finding tables to map those old encodings to Unicode and back. Java is a godsend in this case. Charset.availableCharsets has them all!
What's kinda charming is that TSS also contains text formatting commands. "Make all following text bold! Make it underlined! Now make it both bold and underlined!" Stuff like that.
What's less charming is that TSS is actually a superset (an extension of) the ISO-8859 family, similar to how ISO-8859 is a superset of ASCII. In other words, all ISO-8859-1 (or any other variant) is perfectly valid TSS, but not all TSS is valid ISO-8859-1. This creates a lot of fun meetings with other departments when they query the database and are puzzled as to where those weird characters in their ISO-8859-1 text came from.