26 ms·

This may be tangential, but I think that computer languages should have a different type (and literal notation) for human text (strings that may be read by huma

• bcbrown 13y ago
  In traditional Windows development, the way that was handled was by having human-strings in a resource file (which you could replace with a file in a different language), while the computer-strings were hardcoded constants.
  • anonymfus 13y ago
    Traditional? The same is recommended for .net/WinRT development.
    • bcbrown 13y ago
      I'm not familiar enough with newer Windows development to comment on it, so that's why I qualified my statement.
• tel 13y ago
  Haskell has the different semantics though they share the literal type. In Haskell there's Text for Unicode strings (UTF-16 encoded) and ByteString for, well, byte-strings (fixed vectors of Word8s). The semantics make it hard to confuse them, though if you turn on the OverloadedStrings extension then "foo" is ambiguous. If you don't like that you can just not turn on that extension and then there is no literal syntax for either type and you must convert literal strings (really just lists of characters, [Char]) to either type manually Data.Text.pack "foo" :: Text Data.ByteString.pack "foo" :: ByteString Finally, you must use manual encoding functions to interconvert them Data.Text.Encoding.encodeUtf8 :: Text -> ByteString
  • fulafel 13y ago
    Python has separate bytes and unicode types as well, I think he meant there should be a "black box" type that the program shouldn't try to interpret.
    • tel 13y ago
      I don't think you can really play the same game in Python. The idea of separation of types that occurs in Haskell will really require static typing. It means that entire processing pipelines in your code are completely specialized to one type of string or the other (or generalized to both) with no uncertainty.
• tolmasky 13y ago
  Does Ruby's Symbol type acceptably accomplish this?
  • jerf 13y ago
    Symbols are yet another use case; "I want something as efficient as a small number" (probably 4 bytes nowadays) "but I want to type a nice human name for it, and I don't want to even have to think about the translation". It's neither "text" nor a "binary", it's actually a well-disguised int (or similar type).
    • mratzloff 13y ago
      I love symbols in Ruby. I wish every language had something similar, rather than having me define constants, importing the constants, etc. Just let me create symbols I can pass around easily.
      • mercurial 13y ago
        Debatable. I haven't been doing Ruby for a few years, but when I did, it was mostly used as dictionary keys. So, sure, you didn't need to define your constant somewhere, but on the other hand, you end up a typo away from an invalid lookup. Getting a NameError is a much easier way to locate the origin of the problem.
        • mratzloff 13y ago
          Yes, that is a valid point, and definitely a downside.
  • asolove 13y ago
    But note that ruby's symbols are interned, so if you create lots of them programatically they never get GC'd.
  • asgard1024 13y ago
    For some use cases, symbol types are useful (note that they originally come from Lisp). But not all. For example, in many internet protocols, you need computer strings which are not symbols.
• mekoka 13y ago
  I thought that was the purpose of having binary strings and unicode strings in Python. That's how I've been using them anyhow. Unicode for anything whose text property is relevant (u"this is text") and binary as a simple container of bytes ("bf9@cf0*1!09v$j#x0j").
  • asgard1024 13y ago
    Yes, I think that's a good start, although the literal notation for the computer strings b".." is a bit unwieldy. They also don't coerce when used together, see my comment above.
• bjourne 13y ago
  There is no reason "computer text" couldn't also span the whole unicode character set. I don't think your comment is tangential. But I think the notion that some characters are special or more basic than other characters is a trap and leads to illogical thought. For example, many internet protocols are defined in that way. Ascii only for all commands and responses and then for i18n there are some crazy encoding schemes used (quoted printable, utf7, base64, etc) to encode unicode as ascii. All that goes away if your protocol is standardized on utf8. Then text is text and bytes is bytes.
  • anon1385 13y ago
    Lots of operations people want to do on 'strings' don't make sense on unicode text. There is no 'length'. There is the size in bytes of various encodings and there is the number of code points and there is the number of grapheme clusters. The latter is often what people really want, but to get that you need to know about the fonts being used to render the text. Similarly people want to iterate over a string character by character or take substrings by range but with unicode text that becomes iteration over code points and ranges of code points (unless you go all the way and use a text rendering system to give you grapheme clusters). Code points can be decomposed diacritic marks etc so you can't just blindly insert or change code points at a certain index or take arbitrary substrings without risking breaking the string (you can end up with accents on characters that you didn't intend, or stranded at the end of a string and probably plenty of other types of breakage that I can't even think of). Functionality exists to deal with all this but it's pretty burdensome (e.g. NSString has -rangeOfComposedCharacterSequencesForRange:). That all adds up to a pretty hefty performance penalty as well as potential layering violations (needing to consider fonts and rendering when parsing some protocol if you really are going to treat strings as a sequence of grapheme clusters).
    • bjourne 13y ago
      It certainly is possible to split text into grapheme clusters without involving any font rendering, see: http://www.unicode.org/reports/tr29/ http://www.unicode.org/reports/tr29/ Most text manipulation isn't performance critical, and when it is, you can always implement a fast-path for mostly ascii text.
• cynwoody 13y ago
  In Go, you can name your variables in Chinese† if you want. In Go, strings are immutable UTF-8 byte arrays,†† and the language provides facilities for iterating over them either byte by byte or rune by rune (a rune is an int32, wide enough to hold any unicode character). †http://play.golang.org/p/skyEYhxWg3 http://play.golang.org/p/skyEYhxWg3 ††http://blog.golang.org/strings http://blog.golang.org/strings
  • dbaupp 13y ago
    Does Go provide functions for iterating grapheme by grapheme?
    • MBCook 13y ago
      That's the purpose of the Rune type, I believe. When you iterate over text, it gives you one character at a time, not a fixed number of bytes.
      • pjscott 13y ago
        That's iteration by code point, not by grapheme. For example, the grapheme 'ä' may be represented as two code points: an 'a' and a combining diaeresis. http://en.wikipedia.org/wiki/Code_point http://en.wikipedia.org/wiki/Code_point http://en.wikipedia.org/wiki/Grapheme http://en.wikipedia.org/wiki/Grapheme
    • cynwoody 13y ago
      That's being worked on†. It hasn't made it into the standard package library as far as I know. E.g., there are two ways to write Cañyon City. You can write the ñ as U+00F1 or as an ascii lower-case n followed by a combining tilde (U+0303). The first case results in a single rune, and the second in two runes. Example††. You need additional logic in order to normalize to a canonical representation and realize that the two strings are actually the same. Also, if you are displaying the string, you need to account for the fact that, although the two strings have different byte and rune lengths, they take up exactly the same number of pixels on your display medium. †http://blog.golang.org/normalization http://blog.golang.org/normalization ††http://play.golang.org/p/XJPydELZ6s http://play.golang.org/p/XJPydELZ6s