5 ms·

>> Built-in strings: hopefully UTF-8! > Hopefully just bytes, which trivially allows storage of UTF-8. Yes, it trivially allows, but then you'd need to deal w

by ordu 5y ago

>> Built-in strings: hopefully UTF-8! > Hopefully just bytes, which trivially allows storage of UTF-8. Yes, it trivially allows, but then you'd need to deal with utf8-errors all over the program at runtime. Any method for string would need to be written such a way, that deals with invalid utf8 byte sequences. Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. It gives a lot of edge cases which a programmer must bear in his/her mind constantly, because these edge cases would choose the least expected moment to jump on you. And then you'd need, for example, to invent a way to output an arbitrary byte sequence into a place where only UTF8 is allowed. I, personally, hate it. Like arbitrary byte sequences as a file names, even when I know that no one uses non-utf8 file names, I need to write programs working with file names in such a way, that allows arbitrary byte sequences, and to devise some syntax to output arbitrary byte-sequence into a terminal. Or into a web-page. I see no good reasons to replace strings with arbitrary byte-sequences. If you need arbitrary byte-sequences then you have another abstraction for your task: an array. Much more powerful, because it can be array of bytes, of uint16_t, or of int64_t, or of your own struct. Why to spoil an abstraction of string with an ability to deal with arbitrary byte sequences?

flohofwoe 5y ago
Detecting UTF-8 encoding errors is only needed on the input/output boundary though. This should be solved in a string processing library, not in the language (and all other string processing functions in said library should not produce invalid output strings).
- ordu 5y ago
  Not so easy. String processing library might want to iterate over chars, to do it a code needs to decode UTF8 string, if string is invalid-UTF8, then you'll get UTF8-error while trying to find a substring in a string. Or when trying to get a slice with chars from 5 to 12. Such an error could jump on you unexpectedly in any place of your program. > This should be solved in a string processing library Then a type String also should be defined in a string processing library. Either String follows assumptions of a string processing library, or it doesn't. If it doesn't it makes the task of writing a good processing library much more difficult, and you'll get errors thrown from inside of a string processing library, and you'll need to deal with them.
  - jstimpfle 5y ago
    > String processing library might want to iterate over chars, to do it a code needs to decode UTF8 string UTF-8 was specifically designed such that most code can deal with it byte-by-byte without any decoding step. I've written many parsers for example, they all just read byte-by-byte and special things happen at ASCII characters (such as ';' or '\n'), and it works trivially with UTF-8 inputs, I don't have the care at all. I've also written a text editor with a complicated text rope data structure in it. Do you think I should have made a different text rope for each different text encoding the editor should deal with? No - what my editor does in UTF-8 mode for example, at the visual and editing layer it pulls out data byte-by-byte from the text rope and interprets it as UTF-8. If there is invalid UTF-8 it has to deal with it. But hey, that is the reality of files on a file system - they can contain encoding errors, right? Deal with them! (if you can't simply ignore them). > Or when trying to get a slice with chars from 5 to 12. There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called? If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type. The reality is that data is stored in memory in byte sequences, and that's the representation that a programming language should expose. Everything else is code / libraries.
    - ordu 5y ago
      > I've also written a text editor with a complicated text rope data structure in it. Do you think I should have made a different text rope for each different text encoding the editor should deal with? No, I'd think that you should make a text rope for UTF-8, and then change encoding of text on the boundary. Or, it might be not an UTF8 but some other representation of Unicode, it depends. I see no reason to create a structure for an effective manipulation of strings without choosing a representation of a character at the compile time, because otherwise it would be slower than it might be. The more assumptions about your data you've made at the compile time, the less runtime conditioning you'd need, the faster your code would be. Believe me, I had dealt with different encodings all the time. I'm Russian, and we had three widely used unibyte encodings for a cyrillic, plus different encodings for Unicode. So one had to deal with all of them all the time. The easiest way is to deal internally with Unicode only and to change encoding on the boundaries where your program communicates with an external world. There (on the boundaries) you can deal with errors, like character which cannot be represented in an output encoding (or cannot be represented in an internal one, but if you use Unicode it wouldn't be a problem). You can treat user input as an input in an external encoding and throw errors if she inputs something that cannot be encoded in an output encoding. It works all the time, while making your program able to deal with different internal encodings is a PITA, with errors thrown from the most unexpected places, with a spaghetti code trying to deliver errors to places where these errors could be sensibly dealt with. > There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called? You can use all of them, just pick distinctive names for them, like "char", "glyph", ... and any other you like. But when you did it, you'd want to know where are the boundaries of these things. You'd want to make slices of sequences of these things. If you cannot rely on a validness of underlying UTF8 then you'll be in trouble. > If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type. When I need to work with bytes, I use an array of bytes. Not a string, but an array of bytes. It was hard to grasp after years of experience with a C, but I did managed it at some point. Char is not a byte. Byte is not a char. Array is not a string, string is not an array. When I need an array to deal with bytes, I use an array. When I need a string to deal with characters, I use a string. It is a non-trivial idea for a C-programmer, because all his experience tells him that character and byte is the same thing. So if character is not a byte, then (he reasons) character doesn't exist. If characters as codepoints is a too low abstraction for my task, I can create atop of it another abstraction dealing with glyphs, words, tokens, sentences or something. But the abstraction of codepoints must be a library feature, or I'd be forced to create it myself, to validate UTF8 all over the place, and so on. And if that so, then what the point to have an abstraction of string? If characters as codepoints is a too high abstraction for my task, I can go lower and use an array of bytes. It is really an easy idea, just C as a language tends to confuse people minds by teaching them that char==int8_t. At least my mind was confused and I managed to untangle that mess completely only around my 30th birthday. And several years later I've found that Rust's std is totally differentiate chars/bytes, strings/arrays as I do. I had fallen in love with Rust immediately.
jstimpfle 5y ago
> Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. This stuff is specifically some of the big design wins of these systems. I recommend you to peek over at the Win32 API for example, with its myriads of *A and *W functions, required compiler settings and macro magic to switch between those (even though of course they can't really paper over the difference), then best practice recommendations that have changed multiple times in history, then strange bugs that appear when some invalid data has sneaked into a system that was assumed "pristine"... Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.
- ygra 5y ago
  On Windows, unless you really, really, really have to support Windows 9x/ME you ignore the A functions and exclusively use the W variants. You then also don't need the macro magic to switch, as your code doesn't have to work on both Windows NT and 9x (which is the only reason to use that macro magic). And a difference between "UTF-8" and arbitrary bytes is insofar sensible in that you can perform text operations on the former, but not on the latter. Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
  - jstimpfle 5y ago
    In fact I use exclusively the A variants as far as possible, because anything else is really bad engineering, as I've explained. Even Microsoft has started to acknowledge this, and whereas the A variants had been implemented as wrappers around the W variants before, I heard they started to reverse that and started recommending the A functions. See here for example: https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... If you browse around various documentation, you'll see contradicting statements which variants are recommended, which I take as another sign that the idea of making a distinction on the type level is simply a bad idea. As to the macros, yes, if you give the A or W explicitly, you won't need the macro setting that translates the unspecific names to the A or W variants. And as said, the macros aren't a good idea anyway, as it's still extremely hard to properly abstract the distinction (the types are different sizes!!), so code is generally tied to a specific choice either way. > Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances). It can be stored as a byte stream. Whenever you work with the data, you might need to operate on transformed representations - 32-bit codepoints, or larger combinations, or even more complicated stuff like words, sentences, paragraphs, tags, whatever it needs to do the task as done. This is programming, you transform data to achieve things. What I say is that it's stupid to make a distinction on the type level between things that are entirely the same thing in memory, and that are going to be used for the same things. It's stupid because it unnecessarily create incompatibilities between data and introduces unnecessary "conversions" / copies and requires more code.

by ordu 5y ago

flohofwoe 5y ago

Detecting UTF-8 encoding errors is only needed on the input/output boundary though. This should be solved in a string processing library, not in the language (and all other string processing functions in said library should not produce invalid output strings).

ordu 5y ago
Not so easy. String processing library might want to iterate over chars, to do it a code needs to decode UTF8 string, if string is invalid-UTF8, then you'll get UTF8-error while trying to find a substring in a string. Or when trying to get a slice with chars from 5 to 12. Such an error could jump on you unexpectedly in any place of your program. > This should be solved in a string processing library Then a type String also should be defined in a string processing library. Either String follows assumptions of a string processing library, or it doesn't. If it doesn't it makes the task of writing a good processing library much more difficult, and you'll get errors thrown from inside of a string processing library, and you'll need to deal with them.
- jstimpfle 5y ago
  > String processing library might want to iterate over chars, to do it a code needs to decode UTF8 string UTF-8 was specifically designed such that most code can deal with it byte-by-byte without any decoding step. I've written many parsers for example, they all just read byte-by-byte and special things happen at ASCII characters (such as ';' or '\n'), and it works trivially with UTF-8 inputs, I don't have the care at all. I've also written a text editor with a complicated text rope data structure in it. Do you think I should have made a different text rope for each different text encoding the editor should deal with? No - what my editor does in UTF-8 mode for example, at the visual and editing layer it pulls out data byte-by-byte from the text rope and interprets it as UTF-8. If there is invalid UTF-8 it has to deal with it. But hey, that is the reality of files on a file system - they can contain encoding errors, right? Deal with them! (if you can't simply ignore them). > Or when trying to get a slice with chars from 5 to 12. There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called? If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type. The reality is that data is stored in memory in byte sequences, and that's the representation that a programming language should expose. Everything else is code / libraries.
  - ordu 5y ago
    > I've also written a text editor with a complicated text rope data structure in it. Do you think I should have made a different text rope for each different text encoding the editor should deal with? No, I'd think that you should make a text rope for UTF-8, and then change encoding of text on the boundary. Or, it might be not an UTF8 but some other representation of Unicode, it depends. I see no reason to create a structure for an effective manipulation of strings without choosing a representation of a character at the compile time, because otherwise it would be slower than it might be. The more assumptions about your data you've made at the compile time, the less runtime conditioning you'd need, the faster your code would be. Believe me, I had dealt with different encodings all the time. I'm Russian, and we had three widely used unibyte encodings for a cyrillic, plus different encodings for Unicode. So one had to deal with all of them all the time. The easiest way is to deal internally with Unicode only and to change encoding on the boundaries where your program communicates with an external world. There (on the boundaries) you can deal with errors, like character which cannot be represented in an output encoding (or cannot be represented in an internal one, but if you use Unicode it wouldn't be a problem). You can treat user input as an input in an external encoding and throw errors if she inputs something that cannot be encoded in an output encoding. It works all the time, while making your program able to deal with different internal encodings is a PITA, with errors thrown from the most unexpected places, with a spaghetti code trying to deliver errors to places where these errors could be sensibly dealt with. > There isn't a single definition of "char". What even is a "char"? Is it a byte? Is it a unicode codepoint? Is it any other kind of Unicode combination of codepoints or glyphs or combine sequence and emoji modifiers or whatever all that junk is called? You can use all of them, just pick distinctive names for them, like "char", "glyph", ... and any other you like. But when you did it, you'd want to know where are the boundaries of these things. You'd want to make slices of sequences of these things. If you cannot rely on a validness of underlying UTF8 then you'll be in trouble. > If you need a specific subsequence of some UTF-8 encoded text, use a library that fetches it from the byte storage. There is no point in making a programming language type, because you'll lock in to certain usages, and next thing you'll need is a completely different type. When I need to work with bytes, I use an array of bytes. Not a string, but an array of bytes. It was hard to grasp after years of experience with a C, but I did managed it at some point. Char is not a byte. Byte is not a char. Array is not a string, string is not an array. When I need an array to deal with bytes, I use an array. When I need a string to deal with characters, I use a string. It is a non-trivial idea for a C-programmer, because all his experience tells him that character and byte is the same thing. So if character is not a byte, then (he reasons) character doesn't exist. If characters as codepoints is a too low abstraction for my task, I can create atop of it another abstraction dealing with glyphs, words, tokens, sentences or something. But the abstraction of codepoints must be a library feature, or I'd be forced to create it myself, to validate UTF8 all over the place, and so on. And if that so, then what the point to have an abstraction of string? If characters as codepoints is a too high abstraction for my task, I can go lower and use an array of bytes. It is really an easy idea, just C as a language tends to confuse people minds by teaching them that char==int8_t. At least my mind was confused and I managed to untangle that mess completely only around my 30th birthday. And several years later I've found that Rust's std is totally differentiate chars/bytes, strings/arrays as I do. I had fallen in love with Rust immediately.

jstimpfle 5y ago

> Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. This stuff is specifically some of the big design wins of these systems. I recommend you to peek over at the Win32 API for example, with its myriads of *A and *W functions, required compiler settings and macro magic to switch between those (even though of course they can't really paper over the difference), then best practice recommendations that have changed multiple times in history, then strange bugs that appear when some invalid data has sneaked into a system that was assumed "pristine"... Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.

ygra 5y ago
On Windows, unless you really, really, really have to support Windows 9x/ME you ignore the A functions and exclusively use the W variants. You then also don't need the macro magic to switch, as your code doesn't have to work on both Windows NT and 9x (which is the only reason to use that macro magic). And a difference between "UTF-8" and arbitrary bytes is insofar sensible in that you can perform text operations on the former, but not on the latter. Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
- jstimpfle 5y ago
  In fact I use exclusively the A variants as far as possible, because anything else is really bad engineering, as I've explained. Even Microsoft has started to acknowledge this, and whereas the A variants had been implemented as wrappers around the W variants before, I heard they started to reverse that and started recommending the A functions. See here for example: https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... If you browse around various documentation, you'll see contradicting statements which variants are recommended, which I take as another sign that the idea of making a distinction on the type level is simply a bad idea. As to the macros, yes, if you give the A or W explicitly, you won't need the macro setting that translates the unspecific names to the A or W variants. And as said, the macros aren't a good idea anyway, as it's still extremely hard to properly abstract the distinction (the types are different sizes!!), so code is generally tied to a specific choice either way. > Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances). It can be stored as a byte stream. Whenever you work with the data, you might need to operate on transformed representations - 32-bit codepoints, or larger combinations, or even more complicated stuff like words, sentences, paragraphs, tags, whatever it needs to do the task as done. This is programming, you transform data to achieve things. What I say is that it's stupid to make a distinction on the type level between things that are entirely the same thing in memory, and that are going to be used for the same things. It's stupid because it unnecessarily create incompatibilities between data and introduces unnecessary "conversions" / copies and requires more code.