4 ms·

I am criticizing the part where they allowed [3]u8 to u24 bitCast in the first place. It doesn't make sense logically as u24 is likely not 24 bits in any target

I am criticizing the part where they allowed [3]u8 to u24 bitCast in the first place. It doesn't make sense logically as u24 is likely not 24 bits in any targets let alone portably on every target. Interpreting u24 like it is actually 24 bits sounds like programming in crazy land since it is not 24 bits in any relevant architecture afaik. They didn't allow []u24 with a similar rationale as far as I can remember. I agree with this as someone programming at this level should be able to understand there is no real u24 layout and they should use []u32. Going with the same magical rational they went with here, compiler should generate unaligned u24 loading code when you use []u24 since it is "logically 24 bits"

nyrikki 4mo ago
The ease of dealing with arbitrary bit-width integers and packed structs is actually one of the 'killer features' for me in zig. Zig natively supports arbitrary bit-width integers, the ABI is defined and you could simply think it as a slice of the next larger backing integer. The[3]u8 to u24 bitCast will simply be backed by a 32bit int, using the same ABI. As you have u1 - u65535, sometimes it can be multiple words. The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis. There are very valid use cases when you want to limit utf-8 to U+0000-U+FFFF, and it is valuable if your language allows you to make those decisions. Remember, in zig packed structs are just integers and integers are just a group of logically consecutive bits. Arrays like []u24 do not have the same ABI, arrays are not bit/byte packed, are not universally LSB across archs etc.. The compiler isn't producing unaligned code, don't confuse the abstraction with the concrete implementation. And yes [8]u1 and [8]u8 are exactly the same size and shape, even though they are arrays. My current project is parsing ELF/Macho files, I can easily have zero allocations in my hot path with zig, the same is far more challenging in C, so I am biased, especially with zig allowing methods on structs. And yes, I do use that crazy casting to 0xdeadbeef and other ascii metadata that is in those files. To be clear here, I am not trying to prove you wrong, this is one of the places zig is very different and (IMHO) useful. Especially with streaming data or where you have network ordering etc... It is so nice to only cast what you need to but it does take a little while to wrap your head around how this interacts with buffers which are not your native endianness. At least for me, once I figured out to separate the shape of those data streams from their values it was super useful.
- gucci-on-fleek 4mo ago
  > The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis. I'm not familiar with Zig, so maybe it's doing something weird here, but that doesn't really make sense with Unicode in general. First, the largest Unicode codepoint that will ever be allocated is U+10FFFF [0], which is less than 2^21, so all Unicode characters will fit in a 24-bit integer. Perhaps you're thinking of UCS-2 or UTF-16 without surrogates, which are both 16 bits wide and are limited to the BMP [1] [2] (and therefore don't include most emojis). Second, while the characters needed for most languages lie within the BMP, not all of them do [3], so it isn't really possible to support all languages while excluding emoji, aside from using the Unicode character database to exclude certain categories [4] [5]. [0]: https://www.unicode.org/faq/utf_bom.html#gen0 https://www.unicode.org/faq/utf_bom.html#gen0 [1]: https://www.unicode.org/faq/utf_bom.html#utf16-11 https://www.unicode.org/faq/utf_bom.html#utf16-11 [2]: https://en.wikipedia.org/wiki/Universal_Coded_Character_Set https://en.wikipedia.org/wiki/Universal_Coded_Character_Set [3]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_Multilingual_Plane https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_... [4]: https://www.unicode.org/reports/tr44/tr44-34.html#General_Category_Values https://www.unicode.org/reports/tr44/tr44-34.html#General_Ca... [5]: https://en.wikipedia.org/wiki/Unicode_character_property#General_Category https://en.wikipedia.org/wiki/Unicode_character_property#Gen...
  - nyrikki 4mo ago
    Note the utf-8[0] in my response, the answers are on the pages you linked, but not in the sections you linked, utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8. 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc... There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding. [0] https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8

nyrikki 4mo ago

The ease of dealing with arbitrary bit-width integers and packed structs is actually one of the 'killer features' for me in zig. Zig natively supports arbitrary bit-width integers, the ABI is defined and you could simply think it as a slice of the next larger backing integer. The[3]u8 to u24 bitCast will simply be backed by a 32bit int, using the same ABI. As you have u1 - u65535, sometimes it can be multiple words. The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis. There are very valid use cases when you want to limit utf-8 to U+0000-U+FFFF, and it is valuable if your language allows you to make those decisions. Remember, in zig packed structs are just integers and integers are just a group of logically consecutive bits. Arrays like []u24 do not have the same ABI, arrays are not bit/byte packed, are not universally LSB across archs etc.. The compiler isn't producing unaligned code, don't confuse the abstraction with the concrete implementation. And yes [8]u1 and [8]u8 are exactly the same size and shape, even though they are arrays. My current project is parsing ELF/Macho files, I can easily have zero allocations in my hot path with zig, the same is far more challenging in C, so I am biased, especially with zig allowing methods on structs. And yes, I do use that crazy casting to 0xdeadbeef and other ascii metadata that is in those files. To be clear here, I am not trying to prove you wrong, this is one of the places zig is very different and (IMHO) useful. Especially with streaming data or where you have network ordering etc... It is so nice to only cast what you need to but it does take a little while to wrap your head around how this interacts with buffers which are not your native endianness. At least for me, once I figured out to separate the shape of those data streams from their values it was super useful.

gucci-on-fleek 4mo ago
> The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis. I'm not familiar with Zig, so maybe it's doing something weird here, but that doesn't really make sense with Unicode in general. First, the largest Unicode codepoint that will ever be allocated is U+10FFFF [0], which is less than 2^21, so all Unicode characters will fit in a 24-bit integer. Perhaps you're thinking of UCS-2 or UTF-16 without surrogates, which are both 16 bits wide and are limited to the BMP [1] [2] (and therefore don't include most emojis). Second, while the characters needed for most languages lie within the BMP, not all of them do [3], so it isn't really possible to support all languages while excluding emoji, aside from using the Unicode character database to exclude certain categories [4] [5]. [0]: https://www.unicode.org/faq/utf_bom.html#gen0 https://www.unicode.org/faq/utf_bom.html#gen0 [1]: https://www.unicode.org/faq/utf_bom.html#utf16-11 https://www.unicode.org/faq/utf_bom.html#utf16-11 [2]: https://en.wikipedia.org/wiki/Universal_Coded_Character_Set https://en.wikipedia.org/wiki/Universal_Coded_Character_Set [3]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_Multilingual_Plane https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_... [4]: https://www.unicode.org/reports/tr44/tr44-34.html#General_Category_Values https://www.unicode.org/reports/tr44/tr44-34.html#General_Ca... [5]: https://en.wikipedia.org/wiki/Unicode_character_property#General_Category https://en.wikipedia.org/wiki/Unicode_character_property#Gen...
- nyrikki 4mo ago
  Note the utf-8[0] in my response, the answers are on the pages you linked, but not in the sections you linked, utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8. 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc... There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding. [0] https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8