5 ms·

Persistent virtual memory is the future.

by als0 8y ago

theamk 8y ago
I hope not, I like having readable file formats. When my SymbianOS phone started dying, I found out that there is no way to bulk-export all of my notes -- only way was exporting them one-by-one via laggy user interface. I subsequently spent lots of time analyzing backup file format, and then analyzing notes file format, and then finally extracting the text portions of the notes. Modern apps often use standard libraries for serialization [0], so it is much easier to get to your data back. But persistent virtual memory? We will be back to Symbian world again, but this time, you won't have to reverse engineer unknown on-disk format -- you will have to reverse engineer unknown object model, and chase the pointers to reassemble the data from the tiny pieces. [0] https://jvns.ca/blog/2018/03/31/reverse-engineering-notability-format/ https://jvns.ca/blog/2018/03/31/reverse-engineering-notabili... https://news.ycombinator.com/item?id=16726189 https://news.ycombinator.com/item?id=16726189
- traverseda 8y ago
  Something like capnproto defining interfaces and memory structures in a cross-language way could be one way to mitigate that.
  - theamk 8y ago
    One of the big selling point of "persistent virtual memory" is that you do not need to serialize things to file at all: [0] > Phantom doesn't have files. Well, there are no files in the sense that a developer opens a file handle, writes to it, and closes the file handle. From the user's perspective, things still look familiar – a desktop, directories, and file icons. But a file in Phantom is simply an object whose state is persisted. You don't have to explicitly open it. As long as your program has some kind of reference to that object, all you need to do is call methods on it, and the data is there as you would expect. So in today's OSes, your editor may store data in the complex structure (ropes? ArrayList of Strings? b-tree?), but when saving, it serializes data to the nice format, using capnproto, json, xml, RIFF or something similar. In Phantom, there is no "save" step. If your editor stores text as a list of extents pointing into set of global buffers which contain strings in UCS-4 representation, this is whats gets written to disk, with pointers, vtables and all. Phantom OS magic ensures that pointers and vtables can be restored later. This is nice for application developer (less code), but very bad for third party trying to extract the data -- you either have to run the program and ask it nicely; or reverse engineer the class structure of the program. [0] https://www.theregister.co.uk/2009/02/03/phantom_russian_os/ https://www.theregister.co.uk/2009/02/03/phantom_russian_os/
    - kentonv 8y ago
      Yeah the other classic problem with "Orthogonal Persistence" is that it becomes nearly impossible to upgrade your software. All your existing data is locked to the old version forever (or until some sort of "migration" can be performed -- which requires a way to dump the data, which you may or may not have).
    - cben 8y ago
      Exhibit A: MS Word "fast save" format: https://web.archive.org/web/20160308183811/http://1017.songtrellisopml.com/whatsbeenwroughtusingpiecetables https://web.archive.org/web/20160308183811/http://1017.songt... https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/ https://www.joelonsoftware.com/2008/02/19/why-are-the-micros... From CS perspective, it's an impressive feat of engineering. They used persistent data structures to optimize in-memory operations, and essentially a write-ahead log of these same structures to optimize commits to disk. They also created "OLE" technology in the OS+libraries to support embedding live documents one in another - remarkably not unlike Phantom's goals. The trouble is, they did these in document interchange formats! Office succeeded, and the insane complexity of reverse-engineering its formats actually became a great business "moat". But decades later, after switching the surface structure from binary insanity to XML, it still contains attributes like `autoSpaceLikeWord95` and `footnoteLayoutLikeWW8` :-( http://www.robweir.com/blog/2007/01/how-to-hire-guillaume-portes.html http://www.robweir.com/blog/2007/01/how-to-hire-guillaume-po... A lot has been written assuming malice in OOXML, but you can also argue it'd require unbelievable competence to evolve their codebase to anything other than this! If you never needed pervasive dump/load abstractions, your formats map to 1:1 to your semantics, and the only way to evolve is keeping old semantics around, forever... EDIT: I can't help copy pasting more from the last link: - lineWrapLikeWord6 (Emulate Word 6.0 Line Wrapping for East Asian Text) - mwSmallCaps (Emulate Word 5.x for Macintosh Small Caps Formatting) - shapeLayoutLikeWW8 (Emulate Word 97 Text Wrapping Around Floating Objects) - truncateFontHeightsLikeWP6 (Emulate WordPerfect 6.x Font Height Calculation) - useWord2002TableStyleRules (Emulate Word 2002 Table Style Rules) - useWord97LineBreakRules (Emulate Word 97 East Asian Line Breaking) - wpJustification (Emulate WordPerfect 6.x Paragraph Justification) - shapeLayoutLikeWW8 (Emulate Word 97 Text Wrapping Around Floating Objects)
_0ffh 8y ago
I'm not sure it is a good idea, performance wise. A database application can use differently optimised strategies (data structures, etc.) for memory or disk access. It can't if it doesn't know what's in memory, or on disk.
- als0 8y ago
  > It can't if it doesn't know what's in memory, or on disk. For maximum performance the application can still be aware of the different memory types. You may still have to copy memory around (from volatile to persistent state) but it's possible to avoid the cost of serialisation.
- deleted 8y ago
  [deleted]
- Someone 8y ago
  Presumably, one can still lock down pages in memory (https://www.freebsd.org/cgi/man.cgi?query=mlock https://www.freebsd.org/cgi/man.cgi?query=mlock, http://man7.org/linux/man-pages/man2/mlock.2.html http://man7.org/linux/man-pages/man2/mlock.2.html). So, build default data structures as if everything is swapped out, and add structures designed for use in memory for caching that data. Alternatively, only use structures optimized for in memory use, and trust your OS to keep the active set paged in (easier, but can harm performance if you have much more data than RAM)

Persistent virtual memory is the future.

by als0 8y ago

Persistent virtual memory is the future.

I hope not, I like having readable file formats. When my SymbianOS phone started dying, I found out that there is no way to bulk-export all of my notes -- only way was exporting them one-by-one via laggy user interface. I subsequently spent lots of time analyzing backup file format, and then analyzing notes file format, and then finally extracting the text portions of the notes. Modern apps often use standard libraries for serialization [0], so it is much easier to get to your data back. But persistent virtual memory? We will be back to Symbian world again, but this time, you won't have to reverse engineer unknown on-disk format -- you will have to reverse engineer unknown object model, and chase the pointers to reassemble the data from the tiny pieces. [0] https://jvns.ca/blog/2018/03/31/reverse-engineering-notability-format/ https://jvns.ca/blog/2018/03/31/reverse-engineering-notabili... https://news.ycombinator.com/item?id=16726189 https://news.ycombinator.com/item?id=16726189

traverseda 8y ago
Something like capnproto defining interfaces and memory structures in a cross-language way could be one way to mitigate that.
- theamk 8y ago
  One of the big selling point of "persistent virtual memory" is that you do not need to serialize things to file at all: [0] > Phantom doesn't have files. Well, there are no files in the sense that a developer opens a file handle, writes to it, and closes the file handle. From the user's perspective, things still look familiar – a desktop, directories, and file icons. But a file in Phantom is simply an object whose state is persisted. You don't have to explicitly open it. As long as your program has some kind of reference to that object, all you need to do is call methods on it, and the data is there as you would expect. So in today's OSes, your editor may store data in the complex structure (ropes? ArrayList of Strings? b-tree?), but when saving, it serializes data to the nice format, using capnproto, json, xml, RIFF or something similar. In Phantom, there is no "save" step. If your editor stores text as a list of extents pointing into set of global buffers which contain strings in UCS-4 representation, this is whats gets written to disk, with pointers, vtables and all. Phantom OS magic ensures that pointers and vtables can be restored later. This is nice for application developer (less code), but very bad for third party trying to extract the data -- you either have to run the program and ask it nicely; or reverse engineer the class structure of the program. [0] https://www.theregister.co.uk/2009/02/03/phantom_russian_os/ https://www.theregister.co.uk/2009/02/03/phantom_russian_os/
  - kentonv 8y ago
    Yeah the other classic problem with "Orthogonal Persistence" is that it becomes nearly impossible to upgrade your software. All your existing data is locked to the old version forever (or until some sort of "migration" can be performed -- which requires a way to dump the data, which you may or may not have).
  - cben 8y ago
    Exhibit A: MS Word "fast save" format: https://web.archive.org/web/20160308183811/http://1017.songtrellisopml.com/whatsbeenwroughtusingpiecetables https://web.archive.org/web/20160308183811/http://1017.songt... https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/ https://www.joelonsoftware.com/2008/02/19/why-are-the-micros... From CS perspective, it's an impressive feat of engineering. They used persistent data structures to optimize in-memory operations, and essentially a write-ahead log of these same structures to optimize commits to disk. They also created "OLE" technology in the OS+libraries to support embedding live documents one in another - remarkably not unlike Phantom's goals. The trouble is, they did these in document interchange formats! Office succeeded, and the insane complexity of reverse-engineering its formats actually became a great business "moat". But decades later, after switching the surface structure from binary insanity to XML, it still contains attributes like `autoSpaceLikeWord95` and `footnoteLayoutLikeWW8` :-( http://www.robweir.com/blog/2007/01/how-to-hire-guillaume-portes.html http://www.robweir.com/blog/2007/01/how-to-hire-guillaume-po... A lot has been written assuming malice in OOXML, but you can also argue it'd require unbelievable competence to evolve their codebase to anything other than this! If you never needed pervasive dump/load abstractions, your formats map to 1:1 to your semantics, and the only way to evolve is keeping old semantics around, forever... EDIT: I can't help copy pasting more from the last link: - lineWrapLikeWord6 (Emulate Word 6.0 Line Wrapping for East Asian Text) - mwSmallCaps (Emulate Word 5.x for Macintosh Small Caps Formatting) - shapeLayoutLikeWW8 (Emulate Word 97 Text Wrapping Around Floating Objects) - truncateFontHeightsLikeWP6 (Emulate WordPerfect 6.x Font Height Calculation) - useWord2002TableStyleRules (Emulate Word 2002 Table Style Rules) - useWord97LineBreakRules (Emulate Word 97 East Asian Line Breaking) - wpJustification (Emulate WordPerfect 6.x Paragraph Justification) - shapeLayoutLikeWW8 (Emulate Word 97 Text Wrapping Around Floating Objects)

_0ffh 8y ago

I'm not sure it is a good idea, performance wise. A database application can use differently optimised strategies (data structures, etc.) for memory or disk access. It can't if it doesn't know what's in memory, or on disk.

als0 8y ago
> It can't if it doesn't know what's in memory, or on disk. For maximum performance the application can still be aware of the different memory types. You may still have to copy memory around (from volatile to persistent state) but it's possible to avoid the cost of serialisation.
deleted 8y ago
[deleted]
Someone 8y ago
Presumably, one can still lock down pages in memory (https://www.freebsd.org/cgi/man.cgi?query=mlock https://www.freebsd.org/cgi/man.cgi?query=mlock, http://man7.org/linux/man-pages/man2/mlock.2.html http://man7.org/linux/man-pages/man2/mlock.2.html). So, build default data structures as if everything is swapped out, and add structures designed for use in memory for caching that data. Alternatively, only use structures optimized for in memory use, and trust your OS to keep the active set paged in (easier, but can harm performance if you have much more data than RAM)