5 ms·
Uuuhh, this is an extremely important remark/correction for me, thank you. So far I was thinking as well that any "character" could be saved ("encoded"?) as UT
by zepearl 6y ago
Uuuhh, this is an extremely important remark/correction for me, thank you.
So far I was thinking as well that any "character" could be saved ("encoded"?) as UTF-8 by using max 4 bytes... .
- floatingatoll 6y agoAny UTF-8 code point can be stored in 4 bytes or less, and so as long as you don’t care how many characters your string ends up displaying to the user or try to dissect the string by “characters” somehow, you’ll probably be fine; it’s just a series of valid UTF-8 code points that have meaning you don’t care about at your storage layer. If you’re tokenizing UTF-8 strings or trying to count “displayed” characters, then you also have to watch out for zalgo text that can use unlimited chains of code points and combiners to construct a single phrase out of potentially tens or hundreds of code points. Think it as a recursion limit of sorts. I feel that Unicode and cryptography share the aspect of “don’t try to write your own code to deal with it, because you’re certain to miss something important”.
- zepearl 6y agoThanks! I'll have to review my code... :P