3 ms·

>I want to build a fast data-store Usually this means you want to build things build on arrays. >The problem with mmapped files is that they don't always appe

>I want to build a fast data-store Usually this means you want to build things build on arrays. >The problem with mmapped files is that they don't always appear at the same address i.e. you use arrays and then index into the array instead of using pointers. > So in order to use the standard data-structures, Rust must provide some mechanism to make this possible Well, arrays exist in Rust. Or you're going to have to be more specific about what you're trying to do.

amelius 10y ago
> Or you're going to have to be more specific about what you're trying to do. I want to allocate arrays, maps, etcetera inside the region of a memory mapped file. Then, later, when I mmap the same file to a different region in memory, I want the arrays and maps to still work. That's all.
- steveklabnik 10y ago
  There are several mmap crates, but they won't let you transparently use std::collections on top of them. I imagine this will be possible once the custom allocators work sorts out.
  - dbaupp 10y ago
    Just having custom allocators won't automatically fix the pointer issue: remapping the file later will leave the previously "allocated" pointers pointing into old memory, which may not be where the file was remapped. The allocators and data structures would need to support using non-raw-pointer pointers types, including not even platform-pointer sized types.
    - steveklabnik 10y ago
      Right. Makes sense.
- dbaupp 10y ago
  Do you have examples of how this works in other languages? I would assume they have specialized data structures for this purpose.
  - zvrba 10y ago
    Here: http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/offset_ptr.html http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/o...
- fnord123 10y ago
  As others have explained, it's definitely possible to construct those data structures in Rust and use them from a crate. But you can't just flip a switch to get the standard library collections to work that way. You could probably get pretty far by copy pasting the standard library containers you're interested in and replace Box with something like OffsetBox which uses indices into your memmap arena instead of pointers to any addressable memory. Of course, implementing OffsetBox is an exercise for the reader. :) You would probably have to write some unsafe code to get it to work. But it's not a blocker for a project.
  - amelius 10y ago
    Perhaps a solution would be to introduce the concept of relative pointers. I.e., the pointer becomes a signed value that should be added to the address of the pointer to obtain the address of the location pointed to. Not sure what the implications would be. Of course the standard data-structures would need to support such pointers. After some searching, I found that Microsoft Visual Studio supports something similar, and exactly for the application I had in mind: [1]. [1] https://msdn.microsoft.com/en-us/library/57a97k4e.aspx https://msdn.microsoft.com/en-us/library/57a97k4e.aspx

amelius 10y ago

> Or you're going to have to be more specific about what you're trying to do. I want to allocate arrays, maps, etcetera inside the region of a memory mapped file. Then, later, when I mmap the same file to a different region in memory, I want the arrays and maps to still work. That's all.

steveklabnik 10y ago
There are several mmap crates, but they won't let you transparently use std::collections on top of them. I imagine this will be possible once the custom allocators work sorts out.
- dbaupp 10y ago
  Just having custom allocators won't automatically fix the pointer issue: remapping the file later will leave the previously "allocated" pointers pointing into old memory, which may not be where the file was remapped. The allocators and data structures would need to support using non-raw-pointer pointers types, including not even platform-pointer sized types.
  - steveklabnik 10y ago
    Right. Makes sense.
dbaupp 10y ago
Do you have examples of how this works in other languages? I would assume they have specialized data structures for this purpose.
- zvrba 10y ago
  Here: http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/offset_ptr.html http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/o...
fnord123 10y ago
As others have explained, it's definitely possible to construct those data structures in Rust and use them from a crate. But you can't just flip a switch to get the standard library collections to work that way. You could probably get pretty far by copy pasting the standard library containers you're interested in and replace Box with something like OffsetBox which uses indices into your memmap arena instead of pointers to any addressable memory. Of course, implementing OffsetBox is an exercise for the reader. :) You would probably have to write some unsafe code to get it to work. But it's not a blocker for a project.
- amelius 10y ago
  Perhaps a solution would be to introduce the concept of relative pointers. I.e., the pointer becomes a signed value that should be added to the address of the pointer to obtain the address of the location pointed to. Not sure what the implications would be. Of course the standard data-structures would need to support such pointers. After some searching, I found that Microsoft Visual Studio supports something similar, and exactly for the application I had in mind: [1]. [1] https://msdn.microsoft.com/en-us/library/57a97k4e.aspx https://msdn.microsoft.com/en-us/library/57a97k4e.aspx