7 ms·

> Even Rust (last I checked) defaults to assuming some global one-size-fits-all allocator. You can substitute whatever allocator you want: https://doc.rust-lan

by leshow 6y ago

> Even Rust (last I checked) defaults to assuming some global one-size-fits-all allocator. You can substitute whatever allocator you want: https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html Unless you're talking about some other restriction I'm not aware of.

slimsag 6y ago
In Zig, you can provide a different allocator for a single data structure (or even distinct instances of the same data structure). There is no "global" allocator (what Rust lets you swap out.) This is vastly more powerful, I have used this to tailor an allocator to specific data structures for better performance.
steveklabnik 6y ago
That is what they're referring to, it is a single global allocator, rather than a per-data structure or per-instance one. You can do this in Rust, there's just no abstraction for it. One is coming.
- glandium 6y ago
  I don't know Zig, but it sounds like Zig allows to use arbitrary allocators for anything. The abstraction Rust is getting will only work for things that do account for using arbitrary allocators. Anything that doesn't will end up using the global allocator. That's a significant difference.
  - steveklabnik 6y ago
    That is being backfilled in; Vec already implements it on nightly, IIRC. And really, what you're talking about here is "the standard library data structures," which aren't super likely to be used in firmware anyway. It's a lot easier for ecosystem data structures to add support, after all, they already would choose to call the global allocator, so now they can do either. And it is much easier for them to cut backwards-incompatible changes, if they have to.
    - glandium 6y ago
      I mean, in terms of Rust, it sounds like Zig allows to use any allocator for anything in any crate. Not only structs in std or other crates that explicitly allow a custom allocator. In Rust, and only talking about std, you'd need to change a lot of things to allow e.g. BufWrite, etc. to use a custom allocator. And every crate that uses types that allocate stuff under the hood. But maybe I'm misunderstanding what Zig allows.
      - steveklabnik 6y ago
        You are not misunderstanding what Zig allows, but Rust can do the same thing. https://doc.rust-lang.org/stable/core/alloc/trait.Allocator.html https://doc.rust-lang.org/stable/core/alloc/trait.Allocator.... just isn't stable yet. And it's conventional for it to take this as an argument for everything that needs it in Zig. BufWrite would do it the same as any data structure would, an additional parameter, all the same.
        glandium 6y ago
        I actually did misunderstand. I thought it allowed callers to give an allocator and the callees didn't have to know for it to be used.
        yellowapple 6y ago
        I mean, I'd say you were mostly right, in the sense that the callee doesn't know the implementation details of the passed allocator; it's only aware of the interface (i.e. the struct of function pointers that defines that interface).
    - yellowapple 6y ago
      > which aren't super likely to be used in firmware anyway. Why not? Zig's std library is specifically designed to be usable for freestanding/baremetal targets (e.g. firmware), and the compiler is smart enough to only include the parts of a library (including std) that are actually used. If you do need to reimplement a part of std, you can just... reimplement that part, and import your own implementation instead of the one from std. Unless you're talking purely about Rust?
  - yellowapple 6y ago
    To clarify, it's up to the function being called; the convention set by Zig's standard library is that if a function needs to allocate memory, then the allocator (specifically, a struct of function pointers to an allocator's implementations of realloc and shrink) should be one of the function's arguments. There is of course nothing stopping a function from ignoring this and using the C global allocator if need be (or, as I've done in some experiments, using a C library's custom allocator - in my case that of SQLite). (EDIT: from what I understand, there's technically nothing stopping C from using this sort of strategy, either; a struct of function pointers ain't exactly exotic. It's just a matter of libraries being written with that convention in mind, which doesn't seem to be very common.)
    - glandium 6y ago
      Oh so it has, in fact, the same caveat as Rust's scheme. Edit: I guess the difference is that Zig doesn't have years of not supporting it, making the ecosystem more prone to support it.
    - roca 6y ago
      In a large complex application you are going to want to use the same allocator everywhere, or close to everywhere, and almost all functions may allocate memory directly or indirectly, in which case this Zig convention will require most every function to have a useless parameter. That sounds enraging.
      - tjalfi 6y ago
        It's common to use multiple allocators in some domains; game developers often use a bump allocator[0] which is reset at the end of every frame. [0] https://twitter.com/SebAaltonen/status/1080235671883841541 https://twitter.com/SebAaltonen/status/1080235671883841541
        roca 6y ago
        Thanks, that's a good counterexample.
      - 6y ago
- leshow 6y ago
  What's the use case? I'm trying to think of a situation where you'd want to do that were you might not just make a separate binary that uses the other allocator
  - yellowapple 6y ago
    Zig's docs cover a few different scenarios, but a couple of interest to me at least: - Arena allocators, where your code allocates a chunk of memory and then creates its own allocator just for that chunk; when that chunk gets freed, so does everything in it. Handy for short-lived data. - Using different allocators for different regions of memory makes it trivial to compartmentalize things; you could use the OS allocator (or a straight pointer if you're implementing your own OS) to preallocate chunks of memory, slap allocators on those chunks, and hand those to different components, preventing any given component from bringing down the whole program due to a memory leak. - It's possible to use allocators for verifying code correctness (e.g. detecting memory leaks, testing code under memory exhaustion conditions, etc.).
  - chubot 6y ago
    Custom allocators are extremely common in C and C++ code, and often improve performance over general purpose allocators (though they can also make things worse if you're not careful). The C++ STL has custom allocators (though the initial design was sort of botched; there is a new polymorphic allocator mechanism that aims to fix it IIRC)

by leshow 6y ago

slimsag 6y ago

In Zig, you can provide a different allocator for a single data structure (or even distinct instances of the same data structure). There is no "global" allocator (what Rust lets you swap out.) This is vastly more powerful, I have used this to tailor an allocator to specific data structures for better performance.

steveklabnik 6y ago

That is what they're referring to, it is a single global allocator, rather than a per-data structure or per-instance one. You can do this in Rust, there's just no abstraction for it. One is coming.

glandium 6y ago
I don't know Zig, but it sounds like Zig allows to use arbitrary allocators for anything. The abstraction Rust is getting will only work for things that do account for using arbitrary allocators. Anything that doesn't will end up using the global allocator. That's a significant difference.
- steveklabnik 6y ago
  That is being backfilled in; Vec already implements it on nightly, IIRC. And really, what you're talking about here is "the standard library data structures," which aren't super likely to be used in firmware anyway. It's a lot easier for ecosystem data structures to add support, after all, they already would choose to call the global allocator, so now they can do either. And it is much easier for them to cut backwards-incompatible changes, if they have to.
  - glandium 6y ago
    I mean, in terms of Rust, it sounds like Zig allows to use any allocator for anything in any crate. Not only structs in std or other crates that explicitly allow a custom allocator. In Rust, and only talking about std, you'd need to change a lot of things to allow e.g. BufWrite, etc. to use a custom allocator. And every crate that uses types that allocate stuff under the hood. But maybe I'm misunderstanding what Zig allows.
    - steveklabnik 6y ago
      You are not misunderstanding what Zig allows, but Rust can do the same thing. https://doc.rust-lang.org/stable/core/alloc/trait.Allocator.html https://doc.rust-lang.org/stable/core/alloc/trait.Allocator.... just isn't stable yet. And it's conventional for it to take this as an argument for everything that needs it in Zig. BufWrite would do it the same as any data structure would, an additional parameter, all the same.
      - glandium 6y ago
        I actually did misunderstand. I thought it allowed callers to give an allocator and the callees didn't have to know for it to be used.
        yellowapple 6y ago
        I mean, I'd say you were mostly right, in the sense that the callee doesn't know the implementation details of the passed allocator; it's only aware of the interface (i.e. the struct of function pointers that defines that interface).
  - yellowapple 6y ago
    > which aren't super likely to be used in firmware anyway. Why not? Zig's std library is specifically designed to be usable for freestanding/baremetal targets (e.g. firmware), and the compiler is smart enough to only include the parts of a library (including std) that are actually used. If you do need to reimplement a part of std, you can just... reimplement that part, and import your own implementation instead of the one from std. Unless you're talking purely about Rust?
- yellowapple 6y ago
  To clarify, it's up to the function being called; the convention set by Zig's standard library is that if a function needs to allocate memory, then the allocator (specifically, a struct of function pointers to an allocator's implementations of realloc and shrink) should be one of the function's arguments. There is of course nothing stopping a function from ignoring this and using the C global allocator if need be (or, as I've done in some experiments, using a C library's custom allocator - in my case that of SQLite). (EDIT: from what I understand, there's technically nothing stopping C from using this sort of strategy, either; a struct of function pointers ain't exactly exotic. It's just a matter of libraries being written with that convention in mind, which doesn't seem to be very common.)
  - glandium 6y ago
    Oh so it has, in fact, the same caveat as Rust's scheme. Edit: I guess the difference is that Zig doesn't have years of not supporting it, making the ecosystem more prone to support it.
  - roca 6y ago
    In a large complex application you are going to want to use the same allocator everywhere, or close to everywhere, and almost all functions may allocate memory directly or indirectly, in which case this Zig convention will require most every function to have a useless parameter. That sounds enraging.
    - tjalfi 6y ago
      It's common to use multiple allocators in some domains; game developers often use a bump allocator[0] which is reset at the end of every frame. [0] https://twitter.com/SebAaltonen/status/1080235671883841541 https://twitter.com/SebAaltonen/status/1080235671883841541
      - roca 6y ago
        Thanks, that's a good counterexample.
    - 6y ago
leshow 6y ago
What's the use case? I'm trying to think of a situation where you'd want to do that were you might not just make a separate binary that uses the other allocator
- yellowapple 6y ago
  Zig's docs cover a few different scenarios, but a couple of interest to me at least: - Arena allocators, where your code allocates a chunk of memory and then creates its own allocator just for that chunk; when that chunk gets freed, so does everything in it. Handy for short-lived data. - Using different allocators for different regions of memory makes it trivial to compartmentalize things; you could use the OS allocator (or a straight pointer if you're implementing your own OS) to preallocate chunks of memory, slap allocators on those chunks, and hand those to different components, preventing any given component from bringing down the whole program due to a memory leak. - It's possible to use allocators for verifying code correctness (e.g. detecting memory leaks, testing code under memory exhaustion conditions, etc.).
- chubot 6y ago
  Custom allocators are extremely common in C and C++ code, and often improve performance over general purpose allocators (though they can also make things worse if you're not careful). The C++ STL has custom allocators (though the initial design was sort of botched; there is a new polymorphic allocator mechanism that aims to fix it IIRC)