3 ms·

No you don't. I've written multiple programs that load things instantly off the file system via memory maps. See the fst crate[1], for example, which is designe

No you don't. I've written multiple programs that load things instantly off the file system via memory maps. See the fst crate[1], for example, which is designed to work with memory maps. imdb-rename[2] is a program I wrote that builds a simple IR index on your file system that can then instantly search it by virtue of memory maps. Rust "works badly with memory mapped files" doesn't mean, "Rust can't use memory mapped files." It means, "it is difficult to reconcile Rust's safety story with memory maps." ripgrep for example uses memory maps because they are faster sometimes, and its safety contract[3] is a bit strained. But it works. [1] - https://github.com/BurntSushi/fst/ https://github.com/BurntSushi/fst/ [2] - https://github.com/BurntSushi/imdb-rename https://github.com/BurntSushi/imdb-rename [3] - https://docs.rs/grep-searcher/0.1.7/grep_searcher/struct.MmapChoice.html#method.auto https://docs.rs/grep-searcher/0.1.7/grep_searcher/struct.Mma...

amelius 6y ago
I didn't read your code but one problem I suspect you ran into is that you had to re-invent your container data structures to make them work in a mmapped context.
- burntsushi 6y ago
  No, I didn't. An fst is a compressed data structure, which means you use it in its compressed form without decompressing it first. If you ported the fst crate to C, it would use the same technique. And in C, you have to design your data structures to be mmap friendly anyway. Same deal in Rust. But this is moving the goal posts. This thread started with "you can't do this." But you can. And I have. Multiple times. And I showed you how.
  - amelius 6y ago
    > which means you use it in its compressed form without decompressing it first. So your code operates directly on a block of raw bytes? I can see how that can work with mmap without much problems. My argument was more about structured data (created using the type system), which is a level higher than raw bytes.
    - burntsushi 6y ago
      > So your code operates directly on a block of raw bytes? I can see how that can work with mmap without much problems. Correct. It's a finite state machine. The docs of the crate give links to papers if you want to drill down. > My argument was more about structured data (created using the type system), which is a level higher than raw bytes. Yes. You should be able to do in Rust whatever you would do in C. You can tag your types with `repr(C)` to get a consistent memory layout equivalent to whatever C does. But when you memory map stuff like this, you need to take at least all the same precautions as you would in C. That is, you need to build your data structures to be mmap friendly. The most obvious thing that is problematic for mmap structures like this that is otherwise easy to do is pointer indirection. With that said, this technique is not common in Rust because it requires `unsafe` to do it. And when you use `unsafe`, you want to be sure that it's justified. This is all really besides the point. You'd have the same problems if you read a file into heap memory. The main problem in Rust land with memory maps is that they don't fit into Rust's safety story in an obvious way. But this in and of itself doesn't make them inaccessible to you. It just makes it harder to reason about safety.
      - morty_s 6y ago
        Dang, burntsushi up in the house! Hey, just wanted to say I enjoy your work––I've learned a lot from it. Thank you!
- fractionalhare 6y ago
  It's very tedious to debate with someone who explicitly makes assumptions about something (like code) without having read it, and puts the burden of refuting those assumptions on you...

amelius 6y ago

I didn't read your code but one problem I suspect you ran into is that you had to re-invent your container data structures to make them work in a mmapped context.

burntsushi 6y ago
No, I didn't. An fst is a compressed data structure, which means you use it in its compressed form without decompressing it first. If you ported the fst crate to C, it would use the same technique. And in C, you have to design your data structures to be mmap friendly anyway. Same deal in Rust. But this is moving the goal posts. This thread started with "you can't do this." But you can. And I have. Multiple times. And I showed you how.
- amelius 6y ago
  > which means you use it in its compressed form without decompressing it first. So your code operates directly on a block of raw bytes? I can see how that can work with mmap without much problems. My argument was more about structured data (created using the type system), which is a level higher than raw bytes.
  - burntsushi 6y ago
    > So your code operates directly on a block of raw bytes? I can see how that can work with mmap without much problems. Correct. It's a finite state machine. The docs of the crate give links to papers if you want to drill down. > My argument was more about structured data (created using the type system), which is a level higher than raw bytes. Yes. You should be able to do in Rust whatever you would do in C. You can tag your types with `repr(C)` to get a consistent memory layout equivalent to whatever C does. But when you memory map stuff like this, you need to take at least all the same precautions as you would in C. That is, you need to build your data structures to be mmap friendly. The most obvious thing that is problematic for mmap structures like this that is otherwise easy to do is pointer indirection. With that said, this technique is not common in Rust because it requires `unsafe` to do it. And when you use `unsafe`, you want to be sure that it's justified. This is all really besides the point. You'd have the same problems if you read a file into heap memory. The main problem in Rust land with memory maps is that they don't fit into Rust's safety story in an obvious way. But this in and of itself doesn't make them inaccessible to you. It just makes it harder to reason about safety.
    - morty_s 6y ago
      Dang, burntsushi up in the house! Hey, just wanted to say I enjoy your work––I've learned a lot from it. Thank you!
fractionalhare 6y ago
It's very tedious to debate with someone who explicitly makes assumptions about something (like code) without having read it, and puts the burden of refuting those assumptions on you...