3 ms·
Why can't we just save Rust data-structures in a memory-mapped file? That would make saving and loading much more efficient, and it would also make programming
by amelius 4y ago
Why can't we just save Rust data-structures in a memory-mapped file? That would make saving and loading much more efficient, and it would also make programming much more convenient.
- 0dayz 4y agoI would encourage you to try and make that idea happen.
- habitue 4y agoI mean, you can, if there are no pointers. But then you still need to verify the layout when you reload it etc. Once you solve the various edge cases, this is basically what flatbuffers and capnproto are
- j-pb 4y agoThe information what datastructures are in memory is based on compile time static analysis data, you wouldn't know what you're loading without some additional data annotations. It would also require unreasonably fixing memory layout implementation details between versions.
- amelius 4y agoBut these are things Rust can solve for the programmer! It can add annotations to the data and even provide versioning.
- tialaramex 4y agoYou can do this, but only if you know exactly what that state means. In practice this is almost certainly too fragile to be useful. If you change anything you probably invalidate your understanding of this data and cause chaos. Rust's in memory data structures don't have any meaning on their own, they only have meaning in the context of the running program. This is, in fact, the whole point of the New Type Idiom. If I have two types Minutes(u32) and Years(u32) those are represented in memory the same way, an unsigned 32-bit integer, just four bytes. But because Rust is statically Type Checked, everywhere the programs knows which is which, you can't assign a Minutes to a Years, or a Years to a Minutes, for the same reason you can't assign "Hacker News" to an f64. But this means a 32-bit value in your hypothetical file could be either a Minutes or a Years, and a program loading that file needs to know a priori which it is. Once you start coming up with rules about how you'd do that, you're inventing a serialisation format, and you're exactly back to the problem where your original question arose.
- capableweb 4y ago> You can do this, but only if you know exactly what that state means. In practice this is almost certainly too fragile to be useful. If you change anything you probably invalidate your understanding of this data and cause chaos. I've used the crate `savefile` for this in the past (https://docs.rs/savefile/latest/savefile/ https://docs.rs/savefile/latest/savefile/), which also supports migrating data between versions. So if you have a bunch of on-disk data of a struct and you add a new property, you can handle migrating the data on the fly to the new struct structure. I'm also using `savefile` for saving HN items (comments and stories) to disk in my own HN application: https://codeberg.org/ditzes/ditzes/src/branch/master/src-tauri/src/hn.rs#L30-L47 https://codeberg.org/ditzes/ditzes/src/branch/master/src-tau... and it's very quick to load stuff from disk, haven't hit any bottlenecks really Example: https://ditzes.com/item/35640002 https://ditzes.com/item/35640002 (SpaceX Starship rocket explodes minutes after launch from Texas) has 1223 comments which are individually loaded from disk, takes around 100ms to load from disk, which considering how inefficient and unoptimized the code is, I'm pretty happy with.
- palunon 4y ago> Once you start coming up with rules about how you'd do that, you're inventing a serialisation format, and you're exactly back to the problem where your original question arose. Which is exactly what savefile does.
- amelius 4y agoSorry, but I think you are turning small hurdles into roadblocks in your mind. It is certainly possible to store data structures directly inside files (I think MS Word did this to save documents), but indeed you have to think carefully about type safety but also data alignment, offsetting and such, and perhaps cryptographic signatures to make it secure when needed, which is what library/language designers need to think about to make it easier for programmers to use this stuff. Sure, it won't be adequate in all situations, but that's not what I'm asking for.
- Findecanor 4y agoBesides the issues that others have mentioned (typing, storing nested data-structures, pointers), you would add a requirement on the backing OS and file system to enforce Mandatory File Locking. While Windows enforces it, most Unix systems don't support it at all. Back when Linux had it (removed in 2021), it had to be enabled when a volume was mounted, and that was not default nor was it supported for all file system types. Without it, even if you mmap() the file with the MAP_PRIVATE flag, portions of the file could be overwritten by other programs before you read them.
- mort96 4y agoPeople like to send data around between different computers. Different computers have different endianness and potentially different padding requirements and differently sized 'usize's. And there are the challenges with pointers and data structures others have mentioned. And supporting old files in the event of a Rust ABI change becomes incredibly hard.
- kortex 4y agoWe (the institution of programmers broadly speaking) tried that, and it was a hot mess. It's fine for scratch for a single machine, running a single version of the software. As soon as you involve multiple computers talking or multiple software versions over time, it quickly causes tons of issues. There's a reason most serialization formats are either self-describing, have external schemas (json), or offload state to a specific state store engine.
- dmytrish 4y agoIt's a shame you're being downvoted, it's a perfectly valid question. It is possible to do explicitly with `#[repr(C)]` in Rust. This is still fragile: - `usize` and `isize` are not guaranteed to be portable between machines. - pointers (including vtables in trait objects) are not portable between processes - structs and enums may change in source code - file descriptors, mutexes/futexes and any kind of external resources in general are not portable between processes. Thread-local variables are not even portable between threads. In short: memory layout in general depends on a large amount of ephemeral compiler metadata. It is possible to constrain memory layout to something portable (protobufs, flatbuffers, capnproto, thrift, etc), but this seems more like a serialization format, not an efficient memory layout for a compiled language. I'd say this is a good thing to avoid ossification of memory layouts: if you need to unmarshal something, it is better to mark this explicitly. Although the C ABI can be convenient, it constrains optimizations that compilers are allowed to do.