5 ms·

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like ite

• ekidd 3y ago
  One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points). This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type. For another good alternative to this approach, see Ruby.
  • Animats 3y ago
    > separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Yes. Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as s(5)Hello This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems. Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check. Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as (SMILING FACE)(RED HEART) per the spec.
    • alcover 3y ago
      Greetings, Professor Falken. (SMILING FACE)(RED HEART)
  • anonzzzies 3y ago
    > "Strings are always UTF-8. Anything else is a bug. Deal with it." That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.
• zadokshi 3y ago
  You’re definitely wrong. You’re designing a language that only works for “Americans” by default. Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English? You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols and emojis into your text. I’d suggest that if a language doesn’t have a proper character type, it’s not a useful language for the majority of the world. UTF-8 is a reasonable compromise though.
  • alcover 3y ago
    I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling. My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes. I get your concern though and realize parsing let foo = "<unicode>" would be a problem.
    • zadokshi 3y ago
      Apologies if I come off a bit strong on this one. I deal mostly with non “American” text so it’s a pain point. Unicode is so messy, trying to push handling it on to each individual programmer is not an ideal situation. Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively. Imagine being from a country where a language’s built in Strcmp doesn’t work?
      • alcover 3y ago
        No problem ! > Imagine being from a country where a language’s built in Strcmp doesn’t work? Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library". I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...
      • qalmakka 3y ago
• scatters 3y ago
  It depends on the level and domain of your language. A low-level language can get by with just byte arrays. A mid-level language should probably handle at least some encodings, and provide codepoint access. A high-level language should handle locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!).
  • alcover 3y ago
    It's a dialect of C, so somewhere between low and mid. > handle at least some encodings What else could it be than Utf8 ? > locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!) You made up my mind haha. It won't be high-level.
• tialaramex 3y ago
  People expect Strings. You don't have to provide strings, but you should be aware that people expect them and if you provide something else but don't clearly label it as not strings there's an excellent chance they'll go "Oh, strings" and then they'll complain that it doesn't work the way they expected. If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother. WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).
  • alcover 3y ago
    > If I can write "foo".contains('o') [..] but then you pretend you don't have strings, [..] C has strings By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as strchr("foo", 'o') != NULL
    • tialaramex 3y ago
      You could definitely have C's terrible strings, yes. But it is 2023 so that seems like aiming very low indeed