3 ms·

Sure, it's a low-level language, but my concern is that it doesn't even enforce the encoding? Yes, UTF-8 source code prevents string literals from being invalid

Sure, it's a low-level language, but my concern is that it doesn't even enforce the encoding? Yes, UTF-8 source code prevents string literals from being invalid, but any input from users/sockets/files could be invalid UTF-8. Basically, there's no distinction between a "string" with a known length and a byte array with a known length — the same mistake C and C++ have. Conceptually, strings and byte arrays are different things, even if they can be represented the same way, and a type system could enforce that. And yes, UTF-16, used by Java and C#/.NET is a pain-point as it forces conversion from sockets/files from UTF-8 into UTF-16 so they can be used, then another one when writing back. But that's beside the point when talking about Zig.

brabel 2y ago
Of course it doesn't enforce encoding, it needs to be able to handle invalid data, bad encoding etc. if it wants to be a system language. You can trivially enforce it's valid utf-8 if that's what your application requires by using the `@import("std").unicode` package.
flohofwoe 2y ago
Treating UTF-8 as a different and optional view on a type-agnostic bag of bytes is a feature, not a bug ;) (just look at the mess that Python3 made of that topic) Most of the time, data is just passed around without looking at or caring about its content, and for that it doesn't matter if the data is binary bytes, ascii, code-page encoded, shift-jis, utf-8 or any other format, it only matters at the endpoint when "opening the box". UTF-8 encoding/decoding/validation is handled by the stdlib in Zig, not by the language.
mjburgess 2y ago
On an unrelated note, "string" etymologically just means byte array. Whenever I develop a language, or sketch a relevant data type, I use "Text"
- dontlaugh 2y ago
  Postgres gets this right by making SQL TEXT work efficiently.
adgjlsfhk1 2y ago
that's a good thing actually. if your language restricts strings to valid Unicode, you lose the ability to do things like open files who's path contain invalid Unicode characters.
- SAI_Peregrinus 2y ago
  No, you just lose the ability to treat filenames as strings. They need to be their own type, but you can still open them.

brabel 2y ago

Of course it doesn't enforce encoding, it needs to be able to handle invalid data, bad encoding etc. if it wants to be a system language. You can trivially enforce it's valid utf-8 if that's what your application requires by using the `@import("std").unicode` package.

flohofwoe 2y ago

Treating UTF-8 as a different and optional view on a type-agnostic bag of bytes is a feature, not a bug ;) (just look at the mess that Python3 made of that topic) Most of the time, data is just passed around without looking at or caring about its content, and for that it doesn't matter if the data is binary bytes, ascii, code-page encoded, shift-jis, utf-8 or any other format, it only matters at the endpoint when "opening the box". UTF-8 encoding/decoding/validation is handled by the stdlib in Zig, not by the language.

mjburgess 2y ago

On an unrelated note, "string" etymologically just means byte array. Whenever I develop a language, or sketch a relevant data type, I use "Text"

dontlaugh 2y ago
Postgres gets this right by making SQL TEXT work efficiently.

adgjlsfhk1 2y ago

that's a good thing actually. if your language restricts strings to valid Unicode, you lose the ability to do things like open files who's path contain invalid Unicode characters.

SAI_Peregrinus 2y ago
No, you just lose the ability to treat filenames as strings. They need to be their own type, but you can still open them.