4 ms·

> Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. This stuff is specifically some of the big des

> Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. This stuff is specifically some of the big design wins of these systems. I recommend you to peek over at the Win32 API for example, with its myriads of *A and *W functions, required compiler settings and macro magic to switch between those (even though of course they can't really paper over the difference), then best practice recommendations that have changed multiple times in history, then strange bugs that appear when some invalid data has sneaked into a system that was assumed "pristine"... Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering.

ygra 5y ago
On Windows, unless you really, really, really have to support Windows 9x/ME you ignore the A functions and exclusively use the W variants. You then also don't need the macro magic to switch, as your code doesn't have to work on both Windows NT and 9x (which is the only reason to use that macro magic). And a difference between "UTF-8" and arbitrary bytes is insofar sensible in that you can perform text operations on the former, but not on the latter. Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).
- jstimpfle 5y ago
  In fact I use exclusively the A variants as far as possible, because anything else is really bad engineering, as I've explained. Even Microsoft has started to acknowledge this, and whereas the A variants had been implemented as wrappers around the W variants before, I heard they started to reverse that and started recommending the A functions. See here for example: https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... If you browse around various documentation, you'll see contradicting statements which variants are recommended, which I take as another sign that the idea of making a distinction on the type level is simply a bad idea. As to the macros, yes, if you give the A or W explicitly, you won't need the macro setting that translates the unspecific names to the A or W variants. And as said, the macros aren't a good idea anyway, as it's still extremely hard to properly abstract the distinction (the types are different sizes!!), so code is generally tied to a specific choice either way. > Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances). It can be stored as a byte stream. Whenever you work with the data, you might need to operate on transformed representations - 32-bit codepoints, or larger combinations, or even more complicated stuff like words, sentences, paragraphs, tags, whatever it needs to do the task as done. This is programming, you transform data to achieve things. What I say is that it's stupid to make a distinction on the type level between things that are entirely the same thing in memory, and that are going to be used for the same things. It's stupid because it unnecessarily create incompatibilities between data and introduces unnecessary "conversions" / copies and requires more code.
ordu 5y ago
Firstly you do not need A functions. They are there only for old programs. For really old programs from win9x era. Secondly: if there are worse places than Unix, it doesn't mean that Unix is good. I mean, I love Unix, but Unix sometimes a nightmare to deal with. And kernel believing that "string" means "a byte array with a zero-terminator" one of the worst things of Unix. If one needs an array of variable size, it might do either: struct unsized_array_t { size_t size; uint8_t bytes[]; // being really a bytes[size] } or: struct array_pointer_t { size_t size; uint8_t *ptr; } The second, for example, can easily replace C strings, you only need to pass one more size_t into functions dealing with strings. You can get substrings without modifying string or excessive copying. But then you might wonder why to call this "string" if it is just an array with no compile-time known size. The first one is tricker, you'd need to pass around pointers to it, and slices of it would be a problem. But at least you wouldn't need to scan memory to learn the size of it. Unix strings are not strings but zero-terminated arrays of non-null bytes. It would be obvious if you try another way to deal with a variable size. The Unix-Fathers had an idea how to deal with string, but their idea was proven bad. A lot of functions they invented to deal with their "strings" are now deprecated or even forbidden. Like `gets` for example. I'd hate to use strcpy, I'd better use strncpy, and other functions with `n` inside. But if you started to juggle not just with pointers to strings but with those n's also, you would think of representing strings as structs with an embedded n. And -- viola -- all the Unix's string crap goes through a window. You end with variable sized arrays, and you doesn't need strings anymore. Without losing anything useful. > Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering. I like this one. Maybe you are right after all.
- jstimpfle 5y ago
  > Firstly you do not need A functions. They are there only for old programs. For really old programs from win9x era. See here, it might be that they recognized that choices made in the 90's were wrong: https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... "Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes." > I like this one. Maybe you are right after all. Thanks :-)

ygra 5y ago

On Windows, unless you really, really, really have to support Windows 9x/ME you ignore the A functions and exclusively use the W variants. You then also don't need the macro magic to switch, as your code doesn't have to work on both Windows NT and 9x (which is the only reason to use that macro magic). And a difference between "UTF-8" and arbitrary bytes is insofar sensible in that you can perform text operations on the former, but not on the latter. Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances).

jstimpfle 5y ago
In fact I use exclusively the A variants as far as possible, because anything else is really bad engineering, as I've explained. Even Microsoft has started to acknowledge this, and whereas the A variants had been implemented as wrappers around the W variants before, I heard they started to reverse that and started recommending the A functions. See here for example: https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... If you browse around various documentation, you'll see contradicting statements which variants are recommended, which I take as another sign that the idea of making a distinction on the type level is simply a bad idea. As to the macros, yes, if you give the A or W explicitly, you won't need the macro setting that translates the unspecific names to the A or W variants. And as said, the macros aren't a good idea anyway, as it's still extremely hard to properly abstract the distinction (the types are different sizes!!), so code is generally tied to a specific choice either way. > Unicode cannot be treated as just a byte stream as soon as you want to do something to the content (or just for very narrow circumstances). It can be stored as a byte stream. Whenever you work with the data, you might need to operate on transformed representations - 32-bit codepoints, or larger combinations, or even more complicated stuff like words, sentences, paragraphs, tags, whatever it needs to do the task as done. This is programming, you transform data to achieve things. What I say is that it's stupid to make a distinction on the type level between things that are entirely the same thing in memory, and that are going to be used for the same things. It's stupid because it unnecessarily create incompatibilities between data and introduces unnecessary "conversions" / copies and requires more code.

ordu 5y ago

Firstly you do not need A functions. They are there only for old programs. For really old programs from win9x era. Secondly: if there are worse places than Unix, it doesn't mean that Unix is good. I mean, I love Unix, but Unix sometimes a nightmare to deal with. And kernel believing that "string" means "a byte array with a zero-terminator" one of the worst things of Unix. If one needs an array of variable size, it might do either: struct unsized_array_t { size_t size; uint8_t bytes[]; // being really a bytes[size] } or: struct array_pointer_t { size_t size; uint8_t *ptr; } The second, for example, can easily replace C strings, you only need to pass one more size_t into functions dealing with strings. You can get substrings without modifying string or excessive copying. But then you might wonder why to call this "string" if it is just an array with no compile-time known size. The first one is tricker, you'd need to pass around pointers to it, and slices of it would be a problem. But at least you wouldn't need to scan memory to learn the size of it. Unix strings are not strings but zero-terminated arrays of non-null bytes. It would be obvious if you try another way to deal with a variable size. The Unix-Fathers had an idea how to deal with string, but their idea was proven bad. A lot of functions they invented to deal with their "strings" are now deprecated or even forbidden. Like `gets` for example. I'd hate to use strcpy, I'd better use strncpy, and other functions with `n` inside. But if you started to juggle not just with pointers to strings but with those n's also, you would think of representing strings as structs with an embedded n. And -- viola -- all the Unix's string crap goes through a window. You end with variable sized arrays, and you doesn't need strings anymore. Without losing anything useful. > Making a difference between "UTF-8" and arbitrary byte storage is like racism. It isn't only socially inacceptable, it also creates a massive bureaucratic overhead and requires duplication of implementation efforts. I call it bad engineering. I like this one. Maybe you are right after all.

jstimpfle 5y ago
> Firstly you do not need A functions. They are there only for old programs. For really old programs from win9x era. See here, it might be that they recognized that choices made in the 90's were wrong: https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... "Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes." > I like this one. Maybe you are right after all. Thanks :-)