4 ms·
Should strings be represented as codepoint arrays in memory? In C, for example, should I always decode input and work with that, or convert to utf8?
by kizer 8y ago
Should strings be represented as codepoint arrays in memory? In C, for example, should I always decode input and work with that, or convert to utf8?
- steveklabnik 8y agoIt's always tradeoffs. If by "arrays of codepoints" you mean "each element of the array is a single codepoint", then you get O(1) indexing regarding codepoints, but may use up to 4x the memory as the variable-length encoding.
- bumblebritches5 8y agoWhich is fine when you're doing nothing but a simple operation on a string, then reencoding it. Seriously, how many strings have you ever ran across that are more than 1kb? how many strings have you ran across that are more than 256 codepoints?
- steveklabnik 8y agoSure. It's fine sometimes, and not fine other times. That's why it's called a tradeoff. I see strings larger than that all the time, and care about using low RSS. YMMV.
- nirvdrum 8y agoPeople routinely read text files larger than 1 KB into memory. HTTP responses are often larger than 1 KB. Serialized data (JSON, XML, etc.) is often larger than 1 KB.
- bumblebritches5 8y agoHTTP headers are not anywhere near 1kb of text. the packet's themselves are only 1.5kb.
- nirvdrum 8y agoThe headers are one component of a response. The body is often larger than 1 KB.
- bumblebritches5 8y agoand your whole packet is text, without any compression? I doubt that.
- mromanuk 8y agoI’m currently working with a 2 MB string, representing 150k words, with a structure that is a Packed Trie stored as an array for fast access of Unicode Scalars in Swift
- bumblebritches5 8y agoWhy would you use a string to store that in the first place? if you're editing text, make a structure that contains just words, sentences, paragraphs with smaller more manageable strings in there. There's no reason for a single string to contain that much data.
- Ndymium 8y agoThat would require an array of 32 bit integers (basically UTF-32). It wastes a lot of space for most characters and doesn't help that much with indexing since user perceived characters can still be composed of many codepoints. Different languages take different approaches. Python 3.3+, I believe, looks at the string and chooses ASCII/UCS-2/UCS-4 based on the character that takes the highest amount of bytes [1]. Elixir uses UTF-8 for strings (but you can get the codepoints easily), whereas Erlang uses the list of unicode ints style. [1] https://www.b-list.org/weblog/2017/sep/05/how-python-does-unicode/ https://www.b-list.org/weblog/2017/sep/05/how-python-does-un...
- kizer 8y agoThanks for that link. It is very informative.
- masklinn 8y ago> Should strings be represented as codepoint arrays in memory? No. I think the implementor of Factor's unicode support originally did that but it turns out to not be useful: * it blows up memory usage for ASCII and BMP (4 bytes per codepoint versus 1~3) * this also has impact on CPU caches, lowering the average amount of data you can fit in your cache and work on * it requires a complete conversion of incoming ascii and utf8 data (which only get more and more common as time goes on) rather than just a validation * and because Unicode itself is variable-length (combining codepoints) it's not actually helpful when you're trying to properly manipulate unicode data The only "advantage" of representing strings as codepoint arrays is that you get O(1) access to codepoints which is a terrible idea you should not encourage. UTF-32 internal encoding makes some manipulations very slightly easier, but not enough to matter in the long run, and it encourages bad habits. If you don't need the O(1) access thing for backwards compatibility reasons, don't bother.
- bumblebritches5 8y agoHave you ever written your own unicode library, or are you just parroting what you've heard? I have written my own unicode library, normalizing strings as UTF-32 is MUCH MUCH MUCH easier than trying to do the same in UTF-8 or UTF-16. not to mention casefolding, and it also allows you to design better interfaces, so that the actual algorithm only needs to be implemented once.
- greglindahl 8y agoSounds like your library is the poster-child for: > it blows up memory usage for ASCII and BMP (4 bytes per codepoint versus 1~3)
- bumblebritches5 8y agoFor the microseconds it's in the decoded state, sure, but I'm not dealing with massive strings. Literally all the strings I'm dealing with are under 256 codepoints.
- kizer 8y ago
- bumblebritches5 8y agoYes, it gives you flexibility in using UTF-8 and UTF-16 (which despite being a shitty format, you WILL have to support UTF-16), and also overlong sequences don't matter.
- slrz 8y ago> which despite being a shitty format, you WILL have to support UTF-16 Luckily not. If you absolutely must deal with legacy encodings, feed them through a conversion pass beforehand.
- bumblebritches5 8y agoCan you not read? UTF-16 IS the "legacy" format...
- loeg 8y agoGenerally no. You will know when you need it.