3 ms·

This topic is of great interest to me, do you know if there is any related official or community documentation on using arena allocation (and the problems you m

This topic is of great interest to me, do you know if there is any related official or community documentation on using arena allocation (and the problems you mentioned) in Rust? I've only found https://doc.rust-lang.org/1.1.0/arena/index.html https://doc.rust-lang.org/1.1.0/arena/index.html

steveklabnik 6y ago
Crates like https://crates.io/crates/typed-arena https://crates.io/crates/typed-arena and https://github.com/fitzgen/bumpalo https://github.com/fitzgen/bumpalo are the way you do this in today’s Rust, but what he’s referring to is that types like String manage their own allocations and aren’t yet parameterizable by an allocator. So they’re not super easy to use together. In my experience most of the time you need arenas you’re using your own data structure anyway, but YMMV.
- burnthrow 6y ago
  Thanks for linking those projects and for "yet", which led me to the pages below https://rust-lang.github.io/rfcs/1398-kinds-of-allocators.html https://rust-lang.github.io/rfcs/1398-kinds-of-allocators.ht... https://github.com/rust-lang/rust/issues/32838 https://github.com/rust-lang/rust/issues/32838 https://github.com/rust-lang/wg-allocators https://github.com/rust-lang/wg-allocators
- josephg 6y ago
  > In my experience most of the time you need arenas you’re using your own data structure anyway, but YMMV. That makes sense for video games. Recently I was goofing with cyrus-imap. I wanted to parse the emails out of an mbox file into JSON (JMAP). Parsing an email with cyrus currently does about 5-10k calls to malloc, but the objects are all extremely short lived - they just have to live long enough to parse and then convert to JSON. This is a perfect case for a bump allocator - I'd love to allocate all the parsed email fields into an arena and then clear the whole thing when we move on to the next message. Yes, Cyrus uses a ton of its own internal structs for emails, and they're littered with strings and vectors. (Eg for email headers, lists of email recipients, plain text / HTML message content, etc). Looks like bumpalo will do the job, since it implements its own Box, Vector and String. I understand why, but it seems jarring that I'd need to replace the data types in order to change out the allocator like this. I'm definitely keen for GAT landing if it means bumpalo and friends don't need to reinvent the world to be able to change the allocation strategy. Edit: Oooh Vec::new_in is in nightly! Exciting! https://doc.rust-lang.org/beta/std/vec/struct.Vec.html#method.new_in https://doc.rust-lang.org/beta/std/vec/struct.Vec.html#metho...
wging 6y ago
That is from 1.1.0, which is quite old.

steveklabnik 6y ago

Crates like https://crates.io/crates/typed-arena https://crates.io/crates/typed-arena and https://github.com/fitzgen/bumpalo https://github.com/fitzgen/bumpalo are the way you do this in today’s Rust, but what he’s referring to is that types like String manage their own allocations and aren’t yet parameterizable by an allocator. So they’re not super easy to use together. In my experience most of the time you need arenas you’re using your own data structure anyway, but YMMV.

burnthrow 6y ago
Thanks for linking those projects and for "yet", which led me to the pages below https://rust-lang.github.io/rfcs/1398-kinds-of-allocators.html https://rust-lang.github.io/rfcs/1398-kinds-of-allocators.ht... https://github.com/rust-lang/rust/issues/32838 https://github.com/rust-lang/rust/issues/32838 https://github.com/rust-lang/wg-allocators https://github.com/rust-lang/wg-allocators
josephg 6y ago
> In my experience most of the time you need arenas you’re using your own data structure anyway, but YMMV. That makes sense for video games. Recently I was goofing with cyrus-imap. I wanted to parse the emails out of an mbox file into JSON (JMAP). Parsing an email with cyrus currently does about 5-10k calls to malloc, but the objects are all extremely short lived - they just have to live long enough to parse and then convert to JSON. This is a perfect case for a bump allocator - I'd love to allocate all the parsed email fields into an arena and then clear the whole thing when we move on to the next message. Yes, Cyrus uses a ton of its own internal structs for emails, and they're littered with strings and vectors. (Eg for email headers, lists of email recipients, plain text / HTML message content, etc). Looks like bumpalo will do the job, since it implements its own Box, Vector and String. I understand why, but it seems jarring that I'd need to replace the data types in order to change out the allocator like this. I'm definitely keen for GAT landing if it means bumpalo and friends don't need to reinvent the world to be able to change the allocation strategy. Edit: Oooh Vec::new_in is in nightly! Exciting! https://doc.rust-lang.org/beta/std/vec/struct.Vec.html#method.new_in https://doc.rust-lang.org/beta/std/vec/struct.Vec.html#metho...

wging 6y ago

That is from 1.1.0, which is quite old.