4 ms·
Some big part of the data movement can be eliminated by introducing persistent RAM, i.e. the memristor that never happened unfortunately. When the missing compo
by mojuba 7y ago
Some big part of the data movement can be eliminated by introducing persistent RAM, i.e. the memristor that never happened unfortunately. When the missing component is finally invented I think we'll have an opportinity to revisit some of the core OS concepts, such as a file. Just imagine for a second you have a persistent random-access memory device that doesn't require you to serialize/deserialize things, or move and resolve the binary modules. Your binary module/app will be executed right where it's stored, provided that it was resolved and prepared when it was first copied to your file system. Similarly, a JSON file will be stored as a memory structure rather than serialized JSON (converted to text as necessary, e.g. when viewed in a text editor) Etc. etc.
There are some really interesting changes that can potentially happen when the memristor type of memory becomes available, possibly with its own problems too, but with the huge benefit of moving less data.
- golergka 7y agoThere's already an experimental OS that is built on idea of persistent program memory: http://phantomos.org/ http://phantomos.org/
- mojuba 7y agoThat's very intriguing, thanks!
- wongarsu 7y ago> Similarly, a JSON file will be stored as a memory structure rather than serialized JSON You can do this in C++ right now: mmap a file to a memory region, and just create structs in that region. This has two problems: - normally you want to save at controlled points in time, otherwise you have to worry about recovering from states where some function updated part of the data and then crashed (phone ran out of power etc) - just writing a bunch of internal data structures into a file used to be moderately popular and has great performance, but it's a major headache when you ever update them. You end up implementing a versioning scheme and importers for migrating old files to your new application version. At that point a file format that is designed for data exchange is less headache. In general mmap already offers you a way to treat your disk like memory with decent performance (thanks to caching), and the number of good use cases turned out to be somewhat limited. I doubt just making that faster with new technology will change much.
- jacquesm 7y agoThere is a huge difference between what the GP proposes and your interpretation: the GP talks about a structure that is not just a memory backed copy of the data in the file (which still requires an in memory structure pointing to the bits and pieces to make sense of it, or a painful parse step for every access and an impossibility to write back to it). So the whole structure in its native form rather than the contents of the json file, for instance a graph would reside in memory and could be operated on directly. Hence the 'serialized' in the part that you quoted. Once you serialize it the whole thing becomes hamburger and needs to be parsed again before you can operate on it.
- jayshua 7y agoCouldn't you just mmap a binary final with the same in-memory structure rather than a text file?
- jacquesm 7y agoPointers in your memmapped file would be invalid after the second load. So it can't be the exact same structure that you would normalize use in a memory structure. You could do it with relative offsets if you wanted, that's getting pretty close to pre-heating a cache from a snapshot. That way the file would not contain pointers but you could still traverse it relatively quickly by adding the offsets to the base address of the whole file.
- mojuba 7y agoPointers could be valid if each file in this new OS resided at a specific location say in a 64-bit address space. You could allocate for example 4GB of virtual memory to each file and guarantee that a file will always be found at the same location with all the pointers intact and valid.
- jacquesm 7y agoThat sounds pretty hacky but it could work. Better to do it right and make the whole thing relocatable and the software addressing it aware of that. If you don't do it that way you may end up with some very interesting bugs, such as when your code also gets loaded into a 4GB segment (not all machines are 64 bits), and now all those pointers are valid throughout your memory image. That's bound to lead to confusion. Most CPUs can do base+offset with very little overhead anyway so there would be no or very little gain.
- IshKebab 7y agoThere's 3D XPoint https://en.wikipedia.org/wiki/3D_XPoint https://en.wikipedia.org/wiki/3D_XPoint
- aitchnyu 7y agoMy pet concern is "have you turned it on and off again?" will no longer work. Just like apps have a robust database and a frontend that needs restarting after getting into a bad state, programs for the Persistent RAM generation may need a robust state management system so that frontend state can be rebuilt.
- taffer 7y agoThis could be solved with transactions: If all processing takes place within transactions, then you can always return to a clean state after a crash.
- dmead 7y agoThe memristor is a real thing, just hasn't been implemented yet right?
- magduf 7y agoBasically, yes: some prototypes have been made in labs, but haven't really seen the light of day. Wikipedia has an article about it here: https://en.wikipedia.org/wiki/Memristor https://en.wikipedia.org/wiki/Memristor As noted in the article, HP says that memristor memory may be commercially available by 2018, so I guess we'll see then....
- zitterbewegung 7y agoWe have persistent ram technology . Intel and Micron have made ssd form factor and also DIMM form factor Optane devices. They are based on 3D xpoint. Putting it on an SOC is possible but would be very expensive at this time .