3 ms·

Here’s something I’ve been wondering. Once an application lives in RAM in a RAM disk, does it have to be duplicated into a working area of the RAM or can it be

by xattt 5y ago

Here’s something I’ve been wondering. Once an application lives in RAM in a RAM disk, does it have to be duplicated into a working area of the RAM or can it be manipulated directly? Is the file system-RAM dichotomy a historical anachronism at this point?

tristor 5y ago
Yes, usually it still most be copied, because on-disk data structures are generally historical relics optimized for HDDs and therefore don't map directly to in-memory data structures. Loading from disk to RAM generally isn't just a copy operation, but requires some sort of processing for most applications. Intel was doing some interesting work with Optane in this area, using performant SSDs with large RAM caches as DIMMs, seen by the system as memory, but able to be controlled via drivers to act as disks. There's obviously a place at a certain point to blur the lines. The biggest challenge is RAM is not persistent, disk is persistent, and that implies many things both for data structures as well as for how we interact with it and what sort of constraints we can assume. The nice thing about good RAM disk software is that it's a bit-for-bit representation in memory persisted to disk, which means at least once it's loaded into RAM any processing becomes your only real limiting factor for the copying required to load a file.
- knome 5y ago
  >Yes, usually it still most be copied I don't think this would be true for linux. since linux's page cache is unified across all mounted file systems, and with linux's tmpfs using the page cache directly for storing files, it doesn't make much sense that it would need to copy files for use. if you formatted and mounted some other filesystem into the ram, sure. but tmpfs should avoid any such issue.
flohofwoe 5y ago
At least for games it makes a lot of sense IMHO to have the entire game including static data structs/tables and asset data (geometry, textures, audio, ...) "baked" into a single image that can be directly mapped into memory. You'll still need some "scratch space" for the dynamic game state (because you don't want to duplicate the entire image as savegame files), but for asset data I think it makes a lot of sense and could simplify and speed up asset "loading" dramatically. PS: it would make even more sense to map most asset data directly into GPU memory, don't know how far off affordable 512 GB video cards are though ;)
- hermitdev 5y ago
  ...this is how you end up with file formats like classic XLS. So, careful what you wish for.
  - flohofwoe 5y ago
    Yeah, thus "games" where exchanging data through standard file formats isn't important. There's at least one big downside though: modding wouldn't exactly be trivial with such monolithic game images.
skywal_l 5y ago
Yes because processes are not mapped 1-to-1 to their executable file format. Have a look at this video[0] which explains the structure of an ELF file (the binary format used for executable files in many OSes). The ELF structure basically describes how the process memory should be initialize with data and code. [0] https://www.youtube.com/watch?v=nC1U1LJQL8o https://www.youtube.com/watch?v=nC1U1LJQL8o

by xattt 5y ago

tristor 5y ago

Yes, usually it still most be copied, because on-disk data structures are generally historical relics optimized for HDDs and therefore don't map directly to in-memory data structures. Loading from disk to RAM generally isn't just a copy operation, but requires some sort of processing for most applications. Intel was doing some interesting work with Optane in this area, using performant SSDs with large RAM caches as DIMMs, seen by the system as memory, but able to be controlled via drivers to act as disks. There's obviously a place at a certain point to blur the lines. The biggest challenge is RAM is not persistent, disk is persistent, and that implies many things both for data structures as well as for how we interact with it and what sort of constraints we can assume. The nice thing about good RAM disk software is that it's a bit-for-bit representation in memory persisted to disk, which means at least once it's loaded into RAM any processing becomes your only real limiting factor for the copying required to load a file.

knome 5y ago
>Yes, usually it still most be copied I don't think this would be true for linux. since linux's page cache is unified across all mounted file systems, and with linux's tmpfs using the page cache directly for storing files, it doesn't make much sense that it would need to copy files for use. if you formatted and mounted some other filesystem into the ram, sure. but tmpfs should avoid any such issue.

flohofwoe 5y ago

At least for games it makes a lot of sense IMHO to have the entire game including static data structs/tables and asset data (geometry, textures, audio, ...) "baked" into a single image that can be directly mapped into memory. You'll still need some "scratch space" for the dynamic game state (because you don't want to duplicate the entire image as savegame files), but for asset data I think it makes a lot of sense and could simplify and speed up asset "loading" dramatically. PS: it would make even more sense to map most asset data directly into GPU memory, don't know how far off affordable 512 GB video cards are though ;)

hermitdev 5y ago
...this is how you end up with file formats like classic XLS. So, careful what you wish for.
- flohofwoe 5y ago
  Yeah, thus "games" where exchanging data through standard file formats isn't important. There's at least one big downside though: modding wouldn't exactly be trivial with such monolithic game images.

skywal_l 5y ago

Yes because processes are not mapped 1-to-1 to their executable file format. Have a look at this video[0] which explains the structure of an ELF file (the binary format used for executable files in many OSes). The ELF structure basically describes how the process memory should be initialize with data and code. [0] https://www.youtube.com/watch?v=nC1U1LJQL8o https://www.youtube.com/watch?v=nC1U1LJQL8o