4 ms·
The way I grok the computer science, strings are sequences of zero or more characters input into automata and unicode is a way of encoding text. Text is a seque
by brudgers 10y ago
The way I grok the computer science, strings are sequences of zero or more characters input into automata and unicode is a way of encoding text. Text is a sequence of one or more glyphs input into humans. Thus, for me, the implementation of String in Python 2 is about as sound from a computer science perspective as an implementation can get. By which I mean that in the end all data boils down to bits and clustering the bits as bytes is about as reasonable as alternatives. On the other hand, Python 3 treats strings as text and this necessitates all the overhead of multiple encodings and converting glyphs from one language to another language according to the messy and inconsistent and incomplete rules of human language. For me, it would have been better if Python 3 (and most other languages) had a 'Text' type in addition to the String type.
The problem is sloppy use of language in a domain where usually it is ok but sometimes it isn't.
- Daishiman 10y agoYour definition of a string is wrong. It was fine when C came about and we only had ASCII, but real strings hold none of the implicit assumptions that are held by C. C's strings are just a stream of zero-terminated unsigned bytes. Nothing more, nothing less. Your vision works until you need to get out of the 255 characters available to encode through ASCII. The original way out was to have a bunch of different, incompatible encodings of different widths for different languages. This proved to be a complete disaster, and Unicode has been the answer. Unicode is not easy; nothing in the domain of human text handling is. But its use case is for human-readable output. You can't really do user-facing applications using the C model of strings.
- brudgers 10y agoI am not a fan of C's strings either. Null termination rather than communicating the length as a preamble has been a font of bugs for decades. On the other hand, ASCII is merely a way of interpreting a stream of bits that chunks the bits in into bytes and maps the bytes to characters. Unicode is another way of interpreting a string of bits which also chunks the bits into bytes, but then chunks bytes together before mapping to characters. Both Unicode and ASCII are abstractions built on top of streams of bits largely (since both contain machine instructions such as <BEL>) intended to communicate text primarily and strings (in the computational sense) secondarily (for example as commands to a REST endpoint). For example C has had a wide character type for about 25 years [1] available as an abstraction built on top of strings...like most of C how wide is wide is implementation dependent and explicit 16 bit and 32 bit wide characters are more recently standardized. [1]:
- Groxx 10y agoTreating strings as "a sequence of bytes" is perfectly fine if you never, ever interpret or manipulate the contents. As soon as you want to e.g. limit to 160 characters and suffix with an ellipsis, you run into problems of "what is a character", and you can't even call it a single unicode codepoint. Is "Z̴̛̺͉͙͚̰̔̏ͧͅ" is a single character, or over 10? Can you even call it a single "glyph" since it's formed of many units? Here's the unicode representation: Z\u0314\u030F\u0367\u0334\u031B\u033A\u0349\u0359\u035A\u0330\u0345 Which is a whopping 46 bytes in UTF-8. Or what about unicode flags and their 2-character representation: https://esham.io/2014/06/unicode-flags https://esham.io/2014/06/unicode-flags Strings as a concept as they currently stand are absolute nonsense. A "Text" type might resolve the semantic problems (and I love the name, this is a great idea), but strings are text. Any other use is just abusing the container because it's easier to type "v1.2.3" than to make a "Version(1,2,3)" structure (especially when you have to communicate it across different programs / languages).
- brudgers 10y agoTo me, because texts get encoded into strings (or strings encode texts) they are not the same thing. For example, Base 64 encodes data into a string, but the source is not necessarily a text. That's a separate issue from the way 'string' often gets used in the context of programming...a context in which even the otherwise pedantic seem to lose the faith ('regular expressions' is another one that is closely related). Whether or not Z̴̛̺͉͙͚̰̔̏ͧͅ is a character (in terms of computer science) is a matter of whether or not it is part of the input language which some machine accepts. Which is to say it is no different than whether or not 'HashMap' is part of the input language to a compiler (yes for Java, no for Python).
- Groxx 10y agoBase64 is a text representation of a binary blob of data - it's just a protocol that happens to limit itself to a less-likely-to-be-mangled-by-bad-string-handling-code sequence of bytes. Regexes are a great example - they're text that is parsed into a parsing-engine that can be executed. The text part is just a human-interface protocol over many possible implementations, and importantly, it has a standard. Slightly-varying standards at times, but everything trends towards Perl's version, plus/minus some features. And yes - implementations can choose which variant(s) they support, because they control its interpretation. Programming languages don't have a choice about if "Z̴̛̺͉͙͚̰̔̏ͧͅ" is a character though, if they deal with human-language input and output. Humans have already decided. When it's displayed, it either is or is not, often based on the viewer's language (e.g. `str.lower()` is locale-sensitive, but many programming languages ignore this and only deal with ASCII for English speakers). If the program doesn't understand how it's dealing with this human <-> computer protocol and mangles it, it's just as bad as something that mangles other protocols like TCP/IP, except that humans are occasionally more forgiving. --- edit: I should probably tl;dr this. There is a right way and a wrong way to manipulate human-language text. And it's extremely complicated to do correctly - people are difficult. Shoving it under the rug and ignoring it entirely, as has been done by most people in most languages, is 100% the wrong "solution". Python 3 (or even better, Swift) have taken a step in the right direction to reduce accidents - it's painful because it requires correcting long-standing horrifically-wrong habits.
- jstimpfle 10y agoBut you are aware that python2 has not only bytestrings but also what you call "text"? That these are different types and there are implicit conversions between them? $ python2 >>> len('ä'), 'ä'[0] 2, \xc3 >>> len(u'ä'), u'ä'[0] 1, \xe4 >>> 'ä' == u'ä' True I think it's obvious that there are different levels of looking at the data that we have to deal with. If the implementors feel that the best way to enable different levels is having different representations, so be it. Note that it doesn't require twice the amount of memory if you stream the data.