9 ms·

> Consider, for instance, bitcasting a [2]u8 to a u16. Under the old semantics, the result of this operation depends on the target endian: on big-endian targets

> Consider, for instance, bitcasting a [2]u8 to a u16. Under the old semantics, the result of this operation depends on the target endian: on big-endian targets, the first array element became the 8 most significant bits, whereas on little-endian targets, the first array element became the 8 least significant bits. Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target: This is a huge mistake. You would never expect something like bitCast to do this. I don't understand this approach. Why change something so simple and low level to be complicated and high level? Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does. I think this approach as an idea is bad but at least just add another built-in that implements this higher level idea to not break a simple expectation and current behavior?

tialaramex 4mo ago
> This is a huge mistake. You would never expect something like bitCast to do this. Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.
- dnautics 4mo ago
  @intCast
  - tialaramex 4mo ago
    So, since I don't write Zig I had to go look this up, to save anyone else the bother this is what Rust would call an 'as' cast or C programmers might think of as a value cast, it's going to try to make a value which has a similar meaning but of another type, which may be arbitrarily expensive. What people often want here is a transmute, Rust's core::mem::transmute which changes nothing about the bits except what those bits mean, since the bits didn't change and the machine only has bits anyway this is "free".
    - rllj 4mo ago
      There's `@ptrCast` as mentioned in the post, but I agree that it's not optimal. I would really like a `@transmute` builtin. Alternatively I suppose `extern` unions do that, but again, not quite the same syntactically.
      - tialaramex 4mo ago
        For @ptrCast I also now need to care what the language's pointer provenance model is, and AFAICT the answer is Zig didn't get to that yet. If there's some kind of @transmute then we don't need to explain why the pointer cast worked because we never wrote one so that ducks the question, which is simpler.
nvme0n1p1 4mo ago
You don't need to use @bitCast for the behavior you're talking about. @ptrCast still exists.
- ozgrakkurt 4mo ago
  @ptrCast, > Converts a pointer of one type to a pointer of another type. [1] [1] https://ziglang.org/documentation/master/#toc-ptrCast https://ziglang.org/documentation/master/#toc-ptrCast So it is not the same. You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic instead of using @mempcy for everything
  - nvme0n1p1 4mo ago
    Take the address and deref afterwards, and it's exactly the same. Or to say another way: if you want bits to be reinterpreted raw as if they're in memory, then... put them in memory, then reinterpret them. > You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
    - ozgrakkurt 4mo ago
      > Take the address and deref afterwards, and it's exactly the same. It is significantly worse to take address and deref afterwards. You have to do something like: @as(const u32, @ptrCast(&x)). instead of just @bitCast(x) > Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast. This makes sense except breaking existing code that properly handled endianness by doing a conditional @byteSwap. And what you end up with is a more complicated intrinsic compared to something that reinterprets values with same layout size
      - nvme0n1p1 4mo ago
        > This makes sense except breaking existing code Before Zig hits 1.0, users should expect language changes. Has anyone claimed otherwise? If you need the old thing often enough, you can write a wrapper for it. It's a trivial one-liner, as you've shown.
deleted 4mo ago
[deleted]
boricj 4mo ago
If I understand it correctly, it basically boils down to copying bits from the source to the destination, in order from the least significant bit to the most significant bit. It's not equivalent to C++'s reinterpret_cast. I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.
- tremon 4mo ago
  But doesn't that show why this is a bad idea? If I understand correctly, this code: const MyUnion = packed union { full: u16, bytes: [2]u8, }; const value: u16 = 0x55aa; const in_union: MyUnion = @bitCast(value); const without_union: [2]u8 = @bitCast(value); std.debug.assert(without_union[0] == in_union.bytes[0]); std.debug.assert(without_union[1] == in_union.bytes[1]); ...will now succeed or fail depending on the endianness of the target. That looks like the type of footgun that will bring decades of joy.
  - AlienRobot 4mo ago
    I wonder if packed union also got/will get the same "logical bits" treatment?
    - SkiFire13 4mo ago
      My understanding is that the "logical bits" view breaks down for unions, because the nth logical bit could be at different offsets depending on the union variant that's considered active.
  - peesem 4mo ago
    zig does not allow arrays in packed structs/unions specifically for endianness reasons (there may be other reasons as well but endianness is what i know of)
    - tremon 4mo ago
      Ah, that is useful to know. Is that documented somewhere? From what I can quickly find in the obvious place [0], the only requirement is that "all fields in a packed union must have the same @bitSizeOf" and [2]u8 does satisfy that requirement. [0] https://ziglang.org/documentation/0.16.0/#packed-union https://ziglang.org/documentation/0.16.0/#packed-union
jjmarr 4mo ago
> Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does. The reason u32->u24 casting must be well defined is because some hardware (e.g. many GPUs, microcontrollers) only have floating point multipliers. A 24 bit unsigned integer (stored in a 32 bit register) can be losslessly converted to a 32 bit float by the hardware, multiplied, then converted back. This is much faster than doing 32 bit multiplication in software, however, you still need to tell the compiler about this constraint.
- mathisfun123 4mo ago
  > many GPUs Citation please - every single GPU in the literal world supports integer arithmetic for operating on tid, gid, etc.
  - jmalicki 4mo ago
    While the GP might be technically wrong in a narrow sense, GPUs are built for FP, and that's what you want to be doing if you're using them as accelerators.
    - mathisfun123 4mo ago
      You don't know what you're talking about: an enormous amount of TOPs now runs through quantized (read: integer) kernels. Many GPUs don't have even FP64 or even FP32 support.
      - jmalicki 4mo ago
        EDIT: I was completely wrong, I have mostly worked with GGUF and related quantizations that are LUTs, thank you for correcting me.
        mathisfun123 4mo ago
        > The quantized integer kernels aren't running true integer multiplication, the quantization is it's own thing, they're basically enums not integers ELI-a-GPU-compiler-engineer-working-at-a-major-vendor (because I am). Ie I can pull up the design docs for our ALUs and literally see that you're wrong.
AlienRobot 4mo ago
To me it makes sense. If you don't know what endianness is, it doesn't make sense that a program you write in one programming language works for one target but doesn't work for the other. I think endianness is the footgun that Zig is solving, rather than Zig being the one introducing a footgun when you deal with endianness.
- ozgrakkurt 4mo ago
  > If you don't know what endianness is It is not feasible for someone to write endian portable code in a language like Zig without understanding what endianness is imo. Regardless of how they change @bitCast there will be other cases that break this like doing @ptrCast + @memcpy. Also this breaks currently written code that is endian portable and uses @byteSwap like it is done in most other programming languages that do these things.
kevin_thibedeau 4mo ago
GCC has had __int24 for the AVR backend for some time. Useful for larger integers than int16_t while saving 25% over a 32-bit value. C23 does not mandate padding for _BitInt types. It is wrong to assume that will happen or is the optimal implementation for portable code.
- ozgrakkurt 4mo ago
  Thanks for the context, but what I am criticising is this part: > it became allowed to use @bitCast to reinterpret a [3]u8 as a u24 This cant't make sense unless u24 is defined to be 24bits in the first place. It is just silly to allow something like this. It would make so much more sense to me if they started disallowing this or just even print a deprecation notice for it for one release version. > Useful for larger integers than int16_t while saving 25% over a 32-bit value You can't even do []u24 in zig as far as I can remember and understand anyway so this is only happening in a packed struct context. C doesn't mandate padding but C compilers allow having pointers and arrays of irregular _BitInt types as far as I can understand. In this [1] document, in Abi considerations section, it writes that it is defined to have next-power-of-two layout size. Also here (for RISCV) [2] it seems like it is defined with next-power-of-two layout. Also the document here (for x86_64) defines it similarly [3] [1] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2709.pdf https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2709.pdf [2] https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/300 https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/... [3] https://gitlab.com/x86-psABIs/x86-64-ABI/-/tree/master?ref_type=heads https://gitlab.com/x86-psABIs/x86-64-ABI/-/tree/master?ref_t...
  - dnautics 4mo ago
    > This cant't make sense unless u24 is defined to be 24bits in the first place It's worth remembering that zig is a ~hll that should be platform agnostic. suppose someone built a byte-chip with a 24 bit word. the "new" zig way of doing things will be more portable and slot right in, and support 32- and 16-bit datatypes just fine.
  - scrubs 4mo ago
    I sort of agree ... bit casting from an N width integer into an array of ... woah ... that's too far. It's bitcast not byte-cast which has an implied reinterpretation on a same or smaller word size in the cpu. Once you see that the fact somebody has a u24 in their code is between them and the compiler alone. As others probably noted byte casting (keeping the same endianess) is what unions are for.
samatman 4mo ago
I understand the reaction, but I don't agree. I suggest reading the associated proposal[0] along with the devlog, and having a real think about what's going on here. I'm responding to you saying that you "don't understand" the approach: reasonable, and resembles my initial reaction. I was inclined to agree with you, but what decided it for me is that Zig has another mechanism for "reinterpret bytes". It's exposed on the stdlib as std.mem.asBytes, but this is literally a wrapper for the following: @ptrCast(@alignCast(ptr)); So nothing is lost here: if you need, for whatever reason (and those do exist), to get a raw array of underlying bytes, you absolutely may. Std.mem also has bytesToValue(T, bytes) T, which makes a copy. All the ingredients are there, and this family of mem functions are thin wrappers over builtins, which boil down to pointer casting, dereferencing, and comptime magic. Also worth noting: packed structs in Zig are already defined as logically little-endian: the first field is of low significance, the second is above that, and so on. So this makes `@bitCast` consistent with an existing convention of treating integers as logically little-ended, without regard to how they're actually arrayed in memory. Plus it stands to make low-level bit-twiddling, using oddly-sized integers, optimize better. I like that, especially when what we trade for that is: nothing. Nothing at all, this is a pure win. I'd even guess it's that rare language update which silently fixes buggy code, where someone figured "well, basically everything is little-endian already" (or just didn't think about it), and now that code works properly on big-endian machines. [0]: https://github.com/ziglang/zig/issues/19755 https://github.com/ziglang/zig/issues/19755
audunw 4mo ago
I may be damaged from working on IC hardware design and various weird architectures, but I truly can’t comprehend why you’d think this doesn’t make sense. Yeah, if your architecture doesn’t support 24-bit int it maps to 32-bits. But it also declares that the numbers you’re storing should never be larger than 2^24. It’s about type safety, and also run time checks in safe mode I believe. Bitcasting three bytes to a 24-bit type makes just as much sanse as casting 4 bytes to 32-bit. Theres zero reasons to introduce arbitrary artificial constraints on what you can do based on details of (most of) the underlying architectures, which doesn’t even matter for the operation you’re performing.
- imtringued 4mo ago
  If the architecture supports 3 byte types that means it needs to support 3 byte alignments and their powers 9, 27, 81, etc. The easiest way to support this is to always map every 3-byte read operation to two 2-byte reads and then use multiplexers to recombine it into a 24 bit data type. Of course you could also go crazy and store data in 24 bit blocks in your SRAM. That kind of ruins the 8 bit and 16 bit reads though.
IshKebab 4mo ago
I completely disagree. > As a general rule, the new semantics tend to match the behavior of the old semantics on little-endian targets. They've basically said that bit casting is going to be little endian. This simplifies things for the 100% of people that are on little endian machines, while making the code still work for the 0% of people (rounded to the nearest 0.0000001%) that are using big endian machines.

tialaramex 4mo ago

> This is a huge mistake. You would never expect something like bitCast to do this. Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.

dnautics 4mo ago
@intCast
- tialaramex 4mo ago
  So, since I don't write Zig I had to go look this up, to save anyone else the bother this is what Rust would call an 'as' cast or C programmers might think of as a value cast, it's going to try to make a value which has a similar meaning but of another type, which may be arbitrarily expensive. What people often want here is a transmute, Rust's core::mem::transmute which changes nothing about the bits except what those bits mean, since the bits didn't change and the machine only has bits anyway this is "free".
  - rllj 4mo ago
    There's `@ptrCast` as mentioned in the post, but I agree that it's not optimal. I would really like a `@transmute` builtin. Alternatively I suppose `extern` unions do that, but again, not quite the same syntactically.
    - tialaramex 4mo ago
      For @ptrCast I also now need to care what the language's pointer provenance model is, and AFAICT the answer is Zig didn't get to that yet. If there's some kind of @transmute then we don't need to explain why the pointer cast worked because we never wrote one so that ducks the question, which is simpler.

nvme0n1p1 4mo ago

You don't need to use @bitCast for the behavior you're talking about. @ptrCast still exists.

ozgrakkurt 4mo ago
@ptrCast, > Converts a pointer of one type to a pointer of another type. [1] [1] https://ziglang.org/documentation/master/#toc-ptrCast https://ziglang.org/documentation/master/#toc-ptrCast So it is not the same. You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic instead of using @mempcy for everything
- nvme0n1p1 4mo ago
  Take the address and deref afterwards, and it's exactly the same. Or to say another way: if you want bits to be reinterpreted raw as if they're in memory, then... put them in memory, then reinterpret them. > You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
  - ozgrakkurt 4mo ago
    > Take the address and deref afterwards, and it's exactly the same. It is significantly worse to take address and deref afterwards. You have to do something like: @as(const u32, @ptrCast(&x)). instead of just @bitCast(x) > Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast. This makes sense except breaking existing code that properly handled endianness by doing a conditional @byteSwap. And what you end up with is a more complicated intrinsic compared to something that reinterprets values with same layout size
    - nvme0n1p1 4mo ago
      > This makes sense except breaking existing code Before Zig hits 1.0, users should expect language changes. Has anyone claimed otherwise? If you need the old thing often enough, you can write a wrapper for it. It's a trivial one-liner, as you've shown.

deleted 4mo ago

[deleted]

boricj 4mo ago

If I understand it correctly, it basically boils down to copying bits from the source to the destination, in order from the least significant bit to the most significant bit. It's not equivalent to C++'s reinterpret_cast. I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.

tremon 4mo ago
But doesn't that show why this is a bad idea? If I understand correctly, this code: const MyUnion = packed union { full: u16, bytes: [2]u8, }; const value: u16 = 0x55aa; const in_union: MyUnion = @bitCast(value); const without_union: [2]u8 = @bitCast(value); std.debug.assert(without_union[0] == in_union.bytes[0]); std.debug.assert(without_union[1] == in_union.bytes[1]); ...will now succeed or fail depending on the endianness of the target. That looks like the type of footgun that will bring decades of joy.
- AlienRobot 4mo ago
  I wonder if packed union also got/will get the same "logical bits" treatment?
  - SkiFire13 4mo ago
    My understanding is that the "logical bits" view breaks down for unions, because the nth logical bit could be at different offsets depending on the union variant that's considered active.
- peesem 4mo ago
  zig does not allow arrays in packed structs/unions specifically for endianness reasons (there may be other reasons as well but endianness is what i know of)
  - tremon 4mo ago
    Ah, that is useful to know. Is that documented somewhere? From what I can quickly find in the obvious place [0], the only requirement is that "all fields in a packed union must have the same @bitSizeOf" and [2]u8 does satisfy that requirement. [0] https://ziglang.org/documentation/0.16.0/#packed-union https://ziglang.org/documentation/0.16.0/#packed-union

jjmarr 4mo ago

> Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does. The reason u32->u24 casting must be well defined is because some hardware (e.g. many GPUs, microcontrollers) only have floating point multipliers. A 24 bit unsigned integer (stored in a 32 bit register) can be losslessly converted to a 32 bit float by the hardware, multiplied, then converted back. This is much faster than doing 32 bit multiplication in software, however, you still need to tell the compiler about this constraint.

mathisfun123 4mo ago
> many GPUs Citation please - every single GPU in the literal world supports integer arithmetic for operating on tid, gid, etc.
- jmalicki 4mo ago
  While the GP might be technically wrong in a narrow sense, GPUs are built for FP, and that's what you want to be doing if you're using them as accelerators.
  - mathisfun123 4mo ago
    You don't know what you're talking about: an enormous amount of TOPs now runs through quantized (read: integer) kernels. Many GPUs don't have even FP64 or even FP32 support.
    - jmalicki 4mo ago
      EDIT: I was completely wrong, I have mostly worked with GGUF and related quantizations that are LUTs, thank you for correcting me.
      - mathisfun123 4mo ago
        > The quantized integer kernels aren't running true integer multiplication, the quantization is it's own thing, they're basically enums not integers ELI-a-GPU-compiler-engineer-working-at-a-major-vendor (because I am). Ie I can pull up the design docs for our ALUs and literally see that you're wrong.

AlienRobot 4mo ago

To me it makes sense. If you don't know what endianness is, it doesn't make sense that a program you write in one programming language works for one target but doesn't work for the other. I think endianness is the footgun that Zig is solving, rather than Zig being the one introducing a footgun when you deal with endianness.

ozgrakkurt 4mo ago
> If you don't know what endianness is It is not feasible for someone to write endian portable code in a language like Zig without understanding what endianness is imo. Regardless of how they change @bitCast there will be other cases that break this like doing @ptrCast + @memcpy. Also this breaks currently written code that is endian portable and uses @byteSwap like it is done in most other programming languages that do these things.

kevin_thibedeau 4mo ago

GCC has had __int24 for the AVR backend for some time. Useful for larger integers than int16_t while saving 25% over a 32-bit value. C23 does not mandate padding for _BitInt types. It is wrong to assume that will happen or is the optimal implementation for portable code.

ozgrakkurt 4mo ago
Thanks for the context, but what I am criticising is this part: > it became allowed to use @bitCast to reinterpret a [3]u8 as a u24 This cant't make sense unless u24 is defined to be 24bits in the first place. It is just silly to allow something like this. It would make so much more sense to me if they started disallowing this or just even print a deprecation notice for it for one release version. > Useful for larger integers than int16_t while saving 25% over a 32-bit value You can't even do []u24 in zig as far as I can remember and understand anyway so this is only happening in a packed struct context. C doesn't mandate padding but C compilers allow having pointers and arrays of irregular _BitInt types as far as I can understand. In this [1] document, in Abi considerations section, it writes that it is defined to have next-power-of-two layout size. Also here (for RISCV) [2] it seems like it is defined with next-power-of-two layout. Also the document here (for x86_64) defines it similarly [3] [1] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2709.pdf https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2709.pdf [2] https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/300 https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/... [3] https://gitlab.com/x86-psABIs/x86-64-ABI/-/tree/master?ref_type=heads https://gitlab.com/x86-psABIs/x86-64-ABI/-/tree/master?ref_t...
- dnautics 4mo ago
  > This cant't make sense unless u24 is defined to be 24bits in the first place It's worth remembering that zig is a ~hll that should be platform agnostic. suppose someone built a byte-chip with a 24 bit word. the "new" zig way of doing things will be more portable and slot right in, and support 32- and 16-bit datatypes just fine.
- scrubs 4mo ago
  I sort of agree ... bit casting from an N width integer into an array of ... woah ... that's too far. It's bitcast not byte-cast which has an implied reinterpretation on a same or smaller word size in the cpu. Once you see that the fact somebody has a u24 in their code is between them and the compiler alone. As others probably noted byte casting (keeping the same endianess) is what unions are for.

samatman 4mo ago

I understand the reaction, but I don't agree. I suggest reading the associated proposal[0] along with the devlog, and having a real think about what's going on here. I'm responding to you saying that you "don't understand" the approach: reasonable, and resembles my initial reaction. I was inclined to agree with you, but what decided it for me is that Zig has another mechanism for "reinterpret bytes". It's exposed on the stdlib as std.mem.asBytes, but this is literally a wrapper for the following: @ptrCast(@alignCast(ptr)); So nothing is lost here: if you need, for whatever reason (and those do exist), to get a raw array of underlying bytes, you absolutely may. Std.mem also has bytesToValue(T, bytes) T, which makes a copy. All the ingredients are there, and this family of mem functions are thin wrappers over builtins, which boil down to pointer casting, dereferencing, and comptime magic. Also worth noting: packed structs in Zig are already defined as logically little-endian: the first field is of low significance, the second is above that, and so on. So this makes `@bitCast` consistent with an existing convention of treating integers as logically little-ended, without regard to how they're actually arrayed in memory. Plus it stands to make low-level bit-twiddling, using oddly-sized integers, optimize better. I like that, especially when what we trade for that is: nothing. Nothing at all, this is a pure win. I'd even guess it's that rare language update which silently fixes buggy code, where someone figured "well, basically everything is little-endian already" (or just didn't think about it), and now that code works properly on big-endian machines. [0]: https://github.com/ziglang/zig/issues/19755 https://github.com/ziglang/zig/issues/19755

audunw 4mo ago

I may be damaged from working on IC hardware design and various weird architectures, but I truly can’t comprehend why you’d think this doesn’t make sense. Yeah, if your architecture doesn’t support 24-bit int it maps to 32-bits. But it also declares that the numbers you’re storing should never be larger than 2^24. It’s about type safety, and also run time checks in safe mode I believe. Bitcasting three bytes to a 24-bit type makes just as much sanse as casting 4 bytes to 32-bit. Theres zero reasons to introduce arbitrary artificial constraints on what you can do based on details of (most of) the underlying architectures, which doesn’t even matter for the operation you’re performing.

imtringued 4mo ago
If the architecture supports 3 byte types that means it needs to support 3 byte alignments and their powers 9, 27, 81, etc. The easiest way to support this is to always map every 3-byte read operation to two 2-byte reads and then use multiplexers to recombine it into a 24 bit data type. Of course you could also go crazy and store data in 24 bit blocks in your SRAM. That kind of ruins the 8 bit and 16 bit reads though.

IshKebab 4mo ago

I completely disagree. > As a general rule, the new semantics tend to match the behavior of the old semantics on little-endian targets. They've basically said that bit casting is going to be little endian. This simplifies things for the 100% of people that are on little endian machines, while making the code still work for the 0% of people (rounded to the nearest 0.0000001%) that are using big endian machines.