3 ms·
In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I
by danudey 5y ago
In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough.
On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any special characters in our commit messages).
On Windows systems... oh god. The system treats all text as UTF-16 with whatever byte order, and it took me ages to figure out how to get it to convert the data to UTF-8 before encoding it. Eventually it started working, and it worked for a while until it didn't for whatever reason. I ended up tearing out all the code and just encoding the UTF-16 in base64 and then processing that into UTF-8 on the master where I had access to much saner tools.
Generally speaking, "Unicode" works great in most cases, but when you're dealing with systems with weird or unusual encoding habits, like Windows using UTF-16 or MySQL's "utf8" being limited to three bytes per unicode character instead of four, everything goes out the window and it's the wild west all over again.
- pshc 5y agoThe perils of valuing backwards compatibility above all else… imagine having to use UTF16 in this day and age. Happy 2022!
- deleted 5y ago[deleted]
- BlueTemplar 5y agoFunnily base64 suffers from a related issue that the likes of base58 correct : l and I or O and 0 looking similar or even identical depending on the font !
- dahfizz 5y agoWhy does that matter? When would a human need to read and comprehend base64 encoded data?
- kadoban 5y agoBase58 is used for example for Bitcoin addresses. Being able to type an address from one system to another is a nice property, and it's much less error-prone if you don't have to worry about look-alike characters.
- smitty1e 5y agoWe had to ETL .csv data that must have originated in SQLServer. The utf-16 fact about Windows was apparently unknown to my predecessor. Who wrote some nasty c-language binary to copy the data, knock the upper byte off of each character ahead, and save the now ASCII text to a new file of the mysql load. The encoding='utf-16' argument was all that was needed. For want of a nail. . .
- tialaramex 5y agoI've had to fix this before. A co-worker working with data from a 3rd party supplier had gone "Oh this input data is mangled with stray zero bytes, I'll fix that" and of course that destroys any non-ASCII inputs, eventually I'm told that sometimes the import fails, I investigate, and I realise the "mangled" input is just UTF-16 encoded, conditionally remove the "strip zero bytes" hack and tell the decoder it's UTF-16 and it just works correctly. The "maybe strip null bytes" code lived for years "just in case" after I fixed that because people couldn't believe that's all that was ever "wrong" with the data.
- vbezhenar 5y agoUTF-16 is a simple encoding. It should take a few dozens of LoC to convert to UTF-8. At least if you don’t need extreme performance with AVX, etc.
- saati 5y ago> UTF-16 is a simple encoding This isn't true, for starters it's two encodings UTF-16LE and UTF-16BE.
- bjoli 5y agoMy experience is exactly like what you say. Until it isn't. UTF16 seems like such a neat idea until it meets reality. Mostly because of weird interactions of different libraries, languages and operating systems.