3 ms·
It would be nice if rust could support multiple heaps and tracking their memory usage. If you are working with a lot of data, you need to use some kind of hybr
by jeff-davis 5y ago
It would be nice if rust could support multiple heaps and tracking their memory usage.
If you are working with a lot of data, you need to use some kind of hybrid cache-friendly memory/disk algorithm. For instance, to find the exact number of unique values, you might choose a hashing approach. If the hash table grows large and starts swapping, that will be incredibly slow. Instead, it's better to limit the hash table to a certain memory footprint, and when it grows any larger, you partition any new unique values you find and write them to disk. Then, you just count the members of the hash table, save the count, throw the hash table away, and read one of the partitions. Repeat until all the partitions are gone.
That requires a few important capabilities:
1. Track the memory usage of the hash table. It might be easy to estimate in simple cases, but when you have variable-width types that have their own pointers, etc., it gets very messy. It would be nice to just deep copy values into the hash table's memory pool, and then ask how big the memory pool is now. If it's over the limit, don't accept new unique values, just send them to the disk partitions.
2. Free the heap all at once and release all the memory when you are done with the current batch.
These things are not easy in C, but straightforward if you have a custom allocator that uses its own regions of memory. But in rust, it's pretty hard to do this in a generic way.
- nynx 5y agoRust has custom allocators on the way. They're in nightly currently. You can already set a custom global allocator, but soon you'll be able to specify allocators per collection.
- deleted 5y ago[deleted]
- jeff-davis 5y agoGreat! Do you have some links? I found this: https://doc.rust-lang.org/std/alloc/trait.Allocator.html https://doc.rust-lang.org/std/alloc/trait.Allocator.html but it doesn't look like it allows inspecting the size. I'd also like to know how it works to "move a value in" to a collection, does it actually do a memcpy into the collection's heap? Also, is it really a separate instance of the allocator for the collection? That is, can I wipe all the memory for one hash table in O(1) without touching another hash table that uses the same allocator? EDIT: It would also be nice if there was a way to gracefully handle OOM. Maybe if you could specify a maximum size for some heap, it could be a good way to test how your code fails on OOM (which can still happen in a lot of cases with some kernel configs).
- nynx 5y agoYeah, you'll have to add a method to whatever type you implement `Allocator` for that returns the size. When you call `Vec::push` or `HashMap::insert`, it'll allocate on the heap if there isn't already capacity for it in the already allocated space. If the custom allocator you make just allocates one large space from the global allocator, then you can deallocate the whole thing at once in the `Drop` implementation. There is limited documentation at the moment because the feature is still in flux, but take a look at `Vec::new_in` [0] to get an idea of how a collection can use an allocator. [0]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.new_in https://doc.rust-lang.org/std/vec/struct.Vec.html#method.new...
- deleted 5y ago[deleted]
- steveklabnik 5y ago(These APIs return Result, so an OOM would give an AllocError back)