7 ms·

Resizable structs in Zig

152 points by rvrb 1y ago

rvrb 1y ago
I am the author of this post, let me know if you have any questions or feedback :)
- 90s_dev 1y ago
  [flagged]
- azemetre 1y ago
  I sincerely mean this when I write this: please make more Girls on the Beach albums. I really love Splif Tape, reminds me of Julie Ruin during that timeframe as well (2015ish).
  - rvrb 1y ago
    oh, man! you made my day with this comment. those days are long behind me. we were young, dumb, and inebriated. if you're looking for the bands we were trying to sound like, there was a resurgence of this surf/girl group sound in the early 2010s. look for bands like Shannon and the Clams, Hunx and His Punx, Nobunny, Harlem, Ty Segall, Thee Oh Sees.. basically anyone on the now defunct Burger Records
atmikemikeb 1y ago
I thought about dynamically sized types (DSTs) in zig recently. Was considering writing about it. I came to a different conclusion. Why not use zig's opaque? It's pretty clean at this imo: Less metaprogramming but I think nicer to use in some cases. const Connection = opaque { pub const Header = struct { host_len: usize, // add more static fields or dynamic field lengths here //buff_len: usize, }; pub fn init(a: std.mem.Allocator, args: struct { host: []const u8 }) *@This() { var this = a.allocWithOptions(u8, @sizeOf(Header) + host.len, @alignOf(Header), null); @memcpy(this[@sizeOf(Header)..], host); return this.ptr; } pub fn host(self: *const @This()) []const u8 { const bytes: *u8 = @ptrCast(self); const header: *Header = @ptrCast(self); const data = bytes[@sizeOf(Header)..]; const host = data[0..header.host_len]; return host; } }; going off memory so I expect it to not actually compile, but I've definitely done something like this before.
- rvrb 1y ago
  I would describe this approach as 'intrusive' - you're storing the lengths of the arrays behind the pointer, enforcing a certain layout of the memory being allocated. Because the solution outlined in the article stores the lengths alongside the pointer, instead of behind it, there is room for it to work across an ABI (though it currently does not). It's more like a slice in this way. You could in theory implement your opaque approach using this as a utility to avoid the headache of alignment calculations. For this reason, I think that makes the approach outlined in the article more suitable as a candidate for inclusion in the standard library.
  - atmikemikeb 1y ago
    Yeah I think mine is more about being able to provide a `host()` helper function instead of a `.get(.host)` meta function. It is somewhat boilerplate-y. I think it's really a matter of taste haha. Likely yours would be useful regardless if this is done a lot, since it abstracts some of it, if one wants that.
    - rvrb 1y ago
      I've entertained further expanding this API to expose a comptime generated struct of pointers. From the Connection use-case detailed in the article, it would look something like this: pub fn getPtrs(self: Self) struct { client: *Client, host: []u8, read_buffer: []u8, write_buffer: []u8, } { return .{ client: self.get(.client), host: self.get(.host), read_buffer: self.get(.read_buffer), write_buffer: self.get(.write_buffer), } } I haven't done this because I'm not yet convinced it's worth the added complexity
- ethan_smith 1y ago
  The opaque keyword in Zig doesn't support method definitions - it creates an incomplete type that must be implemented elsewhere, not a type that can have inline methods and fields like your example attempts.
  - 1y ago
mananaysiempre 1y ago
> Zig does not, and will not, have VLAs in the language spec. Instead, you can allocate a slice on the heap. If you want to have the data on the stack, use an array as a bounded backing store, and work with a slice into it[.] Too bad, aligned byte-typed VLAs (and a license to retype them as a struct) are what you need to get stack allocation across ABI boundaries the way Swift does it. (A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.) I guess I’ll have to use something else instead.
- rvrb 1y ago
  I think there may be room to expand this implementation to support such a use case. Right now it enforces an `.auto` layout of the struct provided in order to ensure alignment, but its easy to imagine supporting an `extern struct` with a defined layout. Conceivably, an implementation of this `ResizableStruct` that uses an array buffer as backing rather than a heap allocation, and supports the defined layout of an extern struct, could be used to work across the ABI
- throwawaymaths 1y ago
  you can certainly allocate on the stack (like alloca). you just have to overallocate a compile-time known size and have some sort of fallback mechanism or fail if the size requested exceeds the amount created. moreover since the stack allocator is just an allocator, you can use it with any std (or user) datastructure that takes an allocator.
- AndyKelley 1y ago
  Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call. At a fundamental level, runtime-known stack allocation harms code reusability. Edit: commenters identified 2 more puzzle pieces below, but there's still one that didn't get asked about yet :P
  - Conscat 1y ago
    Does anything stop a user from doing this with inline assembly?
konstantinua00 1y ago
one thing I never understood about VLAs - discussion about them always hits a "can't put it on stack safely" and gets halted, forever why not to make it heap-only type? it seems such a useful addition to type system, why ignore it due to one usecase?
- Out_of_Characte 1y ago
  Because arrays simply do not deal with fragmentation. Yes, you could probaly get decent performance on a modern system that has memory overcommit strategy where you could allocate sparse adress ranges where you would probaly never run out of pointers unless you actually write to your variable array. But its just kind of mediocre and you're better off actually dealing with the stack if you can actually deal with certain fixed sizes.
  - konstantinua00 1y ago
    ...what are you talking about? array-like storage with dynamic size has existed since forever - it's vector. over or undercommitting is a solved problem VLA is the way to bring that into type system, so that it can be it's own variable or struct member, with compiler auto-magic-ing size reading to access members after it
    - Out_of_Characte 1y ago
      > auto-magic-ing size reading to access members after it From the article >we now have everything we need to calculate the size, offset and alignment of every field, regardless of their positioning in the struct. >init to allocate the memory >get to get a pointer to a field >resize to resize the arrays >deinit to free the memory You're now suggesting to do exactly what the article is about without being aware of it.
- ori_b 1y ago
  Those effectively exist. They're called slices.
- uecker 1y ago
deleted 1y ago
[deleted]
quotemstr 1y ago
Zig articles tend to get a little too excited about rediscovering longstanding techniques. The author has described a metaprogramming utility for allocating a contiguous hunk of memory, carving this hunk into fields (in the article's example, a fixed-sized Client header, then some number of bytes for host, then some number of bytes for read_buffer, and then some for write_buffer). I'll acknowledge the syntax is convenient, but 1. we've done this since time immemorial in C. See https://learn.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_record https://learn.microsoft.com/en-us/windows/win32/api/evntcons... 2. you can implement this pattern ergonomically in C++, and even moreso once the C++26 reflection stuff comes online 3. the zig implementation is inefficient. It desugars to const Connection = struct { ptr: [*]u8, lens: struct { host: usize, read_buffer: usize, write_buffer: usize, } } That first pointer is needless indirection and probably a cache miss. You should (unless you have specific performance data showing otherwise) store the sizes in the object header, not in an obese pointer to it. (It's bigger than even a fat pointer.)
- rvrb 1y ago
  You have raised a good discussion point or two, but I am not inclined to engage with them due to the tone you've created with the rest of your comment. Would it be productive to jump into a thread on a Ruby article and puff your feathers about how you've always been able to do this in Perl, and also in Python 4 you can do XYZ? I don't think so. For whatever reason, inevitably in threads on systems languages, someone comes in and postures themselves defensively like this. You might want to reflect on that.
  - quotemstr 1y ago
    [flagged]
    - rvrb 1y ago
      I assure you, I am not the one who appears to be reacting in a hair trigger hypersensitive manner in this thread :)
  - deleted 1y ago
qalmakka 1y ago
The author is literally proposing to implement arrays of variant types
edem 1y ago
Should I try Zig? I wanted to learn something that's more low level than the JVM/Node and i have been contemplating rust and go. zig never occurred to me until now.
- hiccuphippo 1y ago
  I think it's simpler than rust, the basics are easier to learn and will get you a long way before you need the more advanced stuff. Go might be easier because it's garbage collected. If you want something closer to C, but not C itself, Zig is a good choice.

152 points by rvrb 1y ago

rvrb 1y ago

I am the author of this post, let me know if you have any questions or feedback :)

90s_dev 1y ago
[flagged]
azemetre 1y ago
I sincerely mean this when I write this: please make more Girls on the Beach albums. I really love Splif Tape, reminds me of Julie Ruin during that timeframe as well (2015ish).
- rvrb 1y ago
  oh, man! you made my day with this comment. those days are long behind me. we were young, dumb, and inebriated. if you're looking for the bands we were trying to sound like, there was a resurgence of this surf/girl group sound in the early 2010s. look for bands like Shannon and the Clams, Hunx and His Punx, Nobunny, Harlem, Ty Segall, Thee Oh Sees.. basically anyone on the now defunct Burger Records

atmikemikeb 1y ago

I thought about dynamically sized types (DSTs) in zig recently. Was considering writing about it. I came to a different conclusion. Why not use zig's opaque? It's pretty clean at this imo: Less metaprogramming but I think nicer to use in some cases. const Connection = opaque { pub const Header = struct { host_len: usize, // add more static fields or dynamic field lengths here //buff_len: usize, }; pub fn init(a: std.mem.Allocator, args: struct { host: []const u8 }) *@This() { var this = a.allocWithOptions(u8, @sizeOf(Header) + host.len, @alignOf(Header), null); @memcpy(this[@sizeOf(Header)..], host); return this.ptr; } pub fn host(self: *const @This()) []const u8 { const bytes: *u8 = @ptrCast(self); const header: *Header = @ptrCast(self); const data = bytes[@sizeOf(Header)..]; const host = data[0..header.host_len]; return host; } }; going off memory so I expect it to not actually compile, but I've definitely done something like this before.

rvrb 1y ago
I would describe this approach as 'intrusive' - you're storing the lengths of the arrays behind the pointer, enforcing a certain layout of the memory being allocated. Because the solution outlined in the article stores the lengths alongside the pointer, instead of behind it, there is room for it to work across an ABI (though it currently does not). It's more like a slice in this way. You could in theory implement your opaque approach using this as a utility to avoid the headache of alignment calculations. For this reason, I think that makes the approach outlined in the article more suitable as a candidate for inclusion in the standard library.
- atmikemikeb 1y ago
  Yeah I think mine is more about being able to provide a `host()` helper function instead of a `.get(.host)` meta function. It is somewhat boilerplate-y. I think it's really a matter of taste haha. Likely yours would be useful regardless if this is done a lot, since it abstracts some of it, if one wants that.
  - rvrb 1y ago
    I've entertained further expanding this API to expose a comptime generated struct of pointers. From the Connection use-case detailed in the article, it would look something like this: pub fn getPtrs(self: Self) struct { client: *Client, host: []u8, read_buffer: []u8, write_buffer: []u8, } { return .{ client: self.get(.client), host: self.get(.host), read_buffer: self.get(.read_buffer), write_buffer: self.get(.write_buffer), } } I haven't done this because I'm not yet convinced it's worth the added complexity
ethan_smith 1y ago
The opaque keyword in Zig doesn't support method definitions - it creates an incomplete type that must be implemented elsewhere, not a type that can have inline methods and fields like your example attempts.
- 1y ago

mananaysiempre 1y ago

> Zig does not, and will not, have VLAs in the language spec. Instead, you can allocate a slice on the heap. If you want to have the data on the stack, use an array as a bounded backing store, and work with a slice into it[.] Too bad, aligned byte-typed VLAs (and a license to retype them as a struct) are what you need to get stack allocation across ABI boundaries the way Swift does it. (A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.) I guess I’ll have to use something else instead.

rvrb 1y ago
I think there may be room to expand this implementation to support such a use case. Right now it enforces an `.auto` layout of the struct provided in order to ensure alignment, but its easy to imagine supporting an `extern struct` with a defined layout. Conceivably, an implementation of this `ResizableStruct` that uses an array buffer as backing rather than a heap allocation, and supports the defined layout of an extern struct, could be used to work across the ABI
throwawaymaths 1y ago
you can certainly allocate on the stack (like alloca). you just have to overallocate a compile-time known size and have some sort of fallback mechanism or fail if the size requested exceeds the amount created. moreover since the stack allocator is just an allocator, you can use it with any std (or user) datastructure that takes an allocator.
AndyKelley 1y ago
Note that not having runtime-known stack allocations is a key piece of the puzzle in Zig's upcoming async I/O strategy because it allows the compiler to calculate upper bound stack usage for a given function call. At a fundamental level, runtime-known stack allocation harms code reusability. Edit: commenters identified 2 more puzzle pieces below, but there's still one that didn't get asked about yet :P
- Conscat 1y ago
  Does anything stop a user from doing this with inline assembly?

konstantinua00 1y ago

one thing I never understood about VLAs - discussion about them always hits a "can't put it on stack safely" and gets halted, forever why not to make it heap-only type? it seems such a useful addition to type system, why ignore it due to one usecase?

Out_of_Characte 1y ago
Because arrays simply do not deal with fragmentation. Yes, you could probaly get decent performance on a modern system that has memory overcommit strategy where you could allocate sparse adress ranges where you would probaly never run out of pointers unless you actually write to your variable array. But its just kind of mediocre and you're better off actually dealing with the stack if you can actually deal with certain fixed sizes.
- konstantinua00 1y ago
  ...what are you talking about? array-like storage with dynamic size has existed since forever - it's vector. over or undercommitting is a solved problem VLA is the way to bring that into type system, so that it can be it's own variable or struct member, with compiler auto-magic-ing size reading to access members after it
  - Out_of_Characte 1y ago
    > auto-magic-ing size reading to access members after it From the article >we now have everything we need to calculate the size, offset and alignment of every field, regardless of their positioning in the struct. >init to allocate the memory >get to get a pointer to a field >resize to resize the arrays >deinit to free the memory You're now suggesting to do exactly what the article is about without being aware of it.
ori_b 1y ago
Those effectively exist. They're called slices.
uecker 1y ago

deleted 1y ago

[deleted]

quotemstr 1y ago

Zig articles tend to get a little too excited about rediscovering longstanding techniques. The author has described a metaprogramming utility for allocating a contiguous hunk of memory, carving this hunk into fields (in the article's example, a fixed-sized Client header, then some number of bytes for host, then some number of bytes for read_buffer, and then some for write_buffer). I'll acknowledge the syntax is convenient, but 1. we've done this since time immemorial in C. See https://learn.microsoft.com/en-us/windows/win32/api/evntcons/ns-evntcons-event_record https://learn.microsoft.com/en-us/windows/win32/api/evntcons... 2. you can implement this pattern ergonomically in C++, and even moreso once the C++26 reflection stuff comes online 3. the zig implementation is inefficient. It desugars to const Connection = struct { ptr: [*]u8, lens: struct { host: usize, read_buffer: usize, write_buffer: usize, } } That first pointer is needless indirection and probably a cache miss. You should (unless you have specific performance data showing otherwise) store the sizes in the object header, not in an obese pointer to it. (It's bigger than even a fat pointer.)

rvrb 1y ago
You have raised a good discussion point or two, but I am not inclined to engage with them due to the tone you've created with the rest of your comment. Would it be productive to jump into a thread on a Ruby article and puff your feathers about how you've always been able to do this in Perl, and also in Python 4 you can do XYZ? I don't think so. For whatever reason, inevitably in threads on systems languages, someone comes in and postures themselves defensively like this. You might want to reflect on that.
- quotemstr 1y ago
  [flagged]
  - rvrb 1y ago
    I assure you, I am not the one who appears to be reacting in a hair trigger hypersensitive manner in this thread :)
- deleted 1y ago

qalmakka 1y ago

The author is literally proposing to implement arrays of variant types

edem 1y ago

Should I try Zig? I wanted to learn something that's more low level than the JVM/Node and i have been contemplating rust and go. zig never occurred to me until now.

hiccuphippo 1y ago
I think it's simpler than rust, the basics are easier to learn and will get you a long way before you need the more advanced stuff. Go might be easier because it's garbage collected. If you want something closer to C, but not C itself, Zig is a good choice.