4 ms·
But to organize memory you need some idea of how much you are using (e.g. how big the in-memory hash table is), and be able to free it in a reasonable pattern (
by jeff-davis 6y ago
But to organize memory you need some idea of how much you are using (e.g. how big the in-memory hash table is), and be able to free it in a reasonable pattern (e.g. deallocate the whole heap/arena at once). Both things are difficult in rust currently, and memory mapping doesn't help with either.
Memory mapping serves a similar purpose as temp files: a way to tell the OS that it can freely page that out to avoid uncontrolled swapping. But you need to do the organization yourself to keep that access sequential, which is hard to do in rust for the reasons that I mentioned.
If you memory map a hash table and have memory pressure, you'll go into the crazy bad scenario of a page-in for every lookup, which is avoidable with better organization.
I think we are basically in agreement but perhaps my point was not clear.
- CyberDildonics 6y ago> But to organize memory you need some idea of how much you are using When I talk about organizing the data structure I'm talking about how the data is layed out in the memory map and how locality is affected. What I'm saying is not that I think all uses are covered by memory mapping, just that data structure layout combined with memory mapping can probably take care of lots of the majority of scenarios. I did not know that organizing memory layout was difficult in Rust since it is very straight forward in C++.
- jeff-davis 6y agoWhat's difficult in Rust is: (1) knowing how much memory a particular structure (like a hash table) is using (2) managing that memory as a single chunk; e.g. to free the whole thing at once so that you can pick up another chunk that you had previously serialized It's not impossible in rust, it's just that you have to build up all the pieces and none of it will integrate nicely with the rest of the rust code.