9 ms·

Phantom OS: Persistent Operating System

136 points by maydemir 5y ago

qwerty456127 5y ago
> Its primary goal is to provide environment for programs thatsurvive OS reboot. Such an environment greatly simplifies software development Simplifies? I can't even imagine such a program. To me a program is something that starts and ends and can also re-start with a fresh state in case something goes wrong.
- WalterGR 5y ago
  What’s the iOS programming model in this regard?
  - thatjoeoverthr 5y ago
    Something like “crash-only software”; programs are expected to be killed and relaunched. Software should cope with it, as there is no swap space and ideally the user shouldn’t be managing memory. An app may create to some degree the appearance of persistence but it’s not persistence. Users, also, can easily force kill apps. Ongoing background operations like downloads, uploads and some media playback is delegated to the operating system. Such operations continue when your process is killed, and you check up on them when your process starts. I hope it’s not TMI, but the standard library includes atomic file saves.
    - WalterGR 5y ago
      Thank you!
- hsbauauvhabzb 5y ago
  What about some middle ground? I’d like my IDE to remain constant between reboots, my browser to some degree is already (if it crashes, it reopens all current tabs on start, which is effectively the same). I would have thought some clever hacks on an existing kernel would be reliable enough rather than an entirely new OS just for one feature, though.
  - webmobdev 5y ago
    Isn't that what Hibernation offers on Windows?
    - hsbauauvhabzb 5y ago
      Good point. I’ve not even tried to hibernate a Linux machine in 10 years though so I have no idea if it works there
    - timka 5y ago
      Sort of like that but you don't have to wait until all your 16Gb of RAM are written to disk. AFAU, Dmitry found a cheap way of making consistent snapshots of a running system. BTW, there was a thread on HN relatively recently where someone mentioned that the hard part of hibernation is not restoring the RAM state but bringing all hardware into the right state at boot. That's why hibernation has been so problematic on Linux.
  - cdogl 5y ago
    That does sound great, but realistically any application needs to be able to recover from a crash anyway - so even with this support from the OS, you just have a new code path to support. Might be nice for users, but it certainly doesn’t simplify anything for developers.
  - jcelerier 5y ago
    The IDE I use, QtCreator, works like this: if you reboot or it crashes you can just restore the previous session on startup and it will bring you exactly where you were.
  - qwerty456127 5y ago
    Why do we need this in a kernel? Why not just serialize the essentials of the apps states and re-load it after reboot/sign-in? I'd prefer the desktop environment to remember which apps were running including which documents were opened in them and their windows positions and just re-launch everything at start-up. Given how fast does everything cold-boot today thanks to modern SSDs, I wouldn't even use standby/hibernate if this worked this way.
xupybd 5y ago
I guess turning off and on again won't fix it with this operating system.
kstenerud 5y ago
This kind of system will suffer from the ratchet problem. A single bug that negatively impacts state is no longer fixable by rebooting. Instead, you have to format/reinstall. Unintended state becomes permanent, and upgrade paths constrained to the point where you must do intentional damage. I'd also be leery of fragmentation. The project is falling into the "simplification trap", where all it's doing is moving inherent complexity around (which eventually requires users to do a bunch of work-arounds) rather than eliminating it (because inherent complexity can't be eliminated; only needless complications can be eliminated). In theory, this sort of setup would be nice in a perfect world, but in the real world of buggy software and faulty hardware and cosmic rays and failing network connections, it's a disaster waiting to happen.
- stormking 5y ago
  Rebooting does not fix any bugs, it just resets the state. I'm not sure this is really the better solution because many bugs never get fixed because "Have you tried turning it off and on, again?"
  - kstenerud 5y ago
    It is the better solution because it delineates the difference between important data that must be preserved, and not important data that can be thrown out and regenerated (turning it off and on again). Systems that eliminate this difference massively increase the damage surface of your data by forcing both kinds to be treated equally and even intermix. And since software will always have bugs, your fallout damage increases exponentially. It's a lot like naive state saving code that just dumps an in-memory struct to disk: The moment that struct changes (adding/removing/changing a type or size), your load code breaks.
  - thatjoeoverthr 5y ago
    Resetting the state is a necessary step when the state is corrupted. You can’t work around this with “don’t corrupt the state”. So many developers are running the other way: isolate state; reduce dependence on state; eliminate side effects; crash only software; etc. The state doesn’t need to be preserved because it simply isn’t that valuable.
ralferoo 5y ago
I'm not sure this actually solves the real problem that people have. Sure, reboots are annoying and pretty much all current OSes could be improved in this regard. Back in the 90s, it's interesting that SunOS supported live kernel upgrades, so at the same time the kernel on disk was updated, the running system was also patched so it would act the same way as-if rebooted, but without any disruption to the live system. However, in the current days where rebooting is seen as normal for an OS upgrade, but most of the time people just put their computer to sleep, the main problem is that all the network interfaces are effectively useless when the system wakes up because most connections will have long since timed out due to lack of connection acknowledgement packets. In such a case, systems will have to tear-down and re-create a lot of state anyway. The actual issue this seems to solve, that of saving memory state and restoring again, is already solved for most use cases except OS upgrades by using sleep mode. But the hard case about network connections is still unsolved for this system, and by solving the problems it does solve with yet another VM-based environment it'll probably be doomed to obscurity unless common existing applications and virtual machines can easily be made to run on-top of it.
- josephg 5y ago
  > all the network interfaces are effectively useless when the system wakes up This is - and probably always will be a problem with networked software. You should program as if TCP connections could drop at any time, for any reason. Browser tabs get backgrounded. Laptops go to sleep. Cell phones roam, and go into tunnels. Servers have temporary net splits. Most high level tasks can be retried safely. Nonces and things can be used to safely retry almost anything else. I miss the simplicity of IRC, the way slack and discord smoothly transition between online and offline states is graceful and intuitive. That is how almost all software should behave.
- xvilka 5y ago
  It's even more essential in IoT and mobile devices, where reboot is often necessary, sometimes just to restore broken connection, like modem. Nobody works now on stable software, everyone tries to ship it as fast as possible. It's not necessarily bad thing, but it is certainly orthogonal to the persistence.
- 5y ago
Findecanor 5y ago
I wonder how much wear on the system disk is caused unnecessarily by taking continuous snapshots. I think that you would like to limit the actual data to persist to the data that needs to be be persisted — that which can't be recreated quickly, purely from other objects.
mahesh-hegde 5y ago
I remember reading about another, pretty old, microkernel OS which had persistent processes. It was probably a capability based OS. Anyone know what that was?
- Findecanor 5y ago
  You are probably thinking of EROS <https://en.wikipedia.org/wiki/EROS_(microkernel) https://en.wikipedia.org/wiki/EROS_(microkernel)> But there have been several others also based on similar ideas, such as Mungi. In recent years, there is Twizzler, intended for NVRAM. <https://www.youtube.com/watch?v=0Ix5DYKxzLI https://www.youtube.com/watch?v=0Ix5DYKxzLI>
  - mahesh-hegde 5y ago
    Yeah it was EROS.
jcun4128 5y ago
Ooh interesting, probably unrelated but I was thinking a GAI would use one of these at its core eg. "it can't die". For the self/state aspect. Also a criteria is code that modify itself without recompiling. This also considers a power source like a nuclear battery or something that will last a long time or alternate in the low power state/ambient or whatever energy.
t43562 5y ago
I studied some persistent object store databases in the past - which seemed like an incredibly good idea and I think the AS400 used such a database and was very popular. It removed the need for a filesystem or any of the usual patterns for retrieving data like SQL so a whole class of programming that people think of as "normal" today just vaporised. Persistent programs seem like a logical-ish next step but I wish the first step could have been taken because it was very nice to program in.
pjmlp 5y ago
I like the ideas in Phantom, even considering the negative comments, it is refreshing to see people having a go at OS design that isn't just blindly making yet another UNIX clone, because apparently we can't get enough of them.
- AnIdiotOnTheNet 5y ago
  Indeed. So many people seem to be brought up with an almost religious indoctrination that UNIX[0] is the end-all be-all of OS design. [0] By which of course we mean Linux.
  - anthk 5y ago
    - GNU's Not Unix. - Persistent OS'es are a thing since BOOTP boots for example with OpenBSD or alpine on "diskless" mode. Thank Unix for that for being so simple to implement.
    - AnIdiotOnTheNet 5y ago
      Are you sure you know what 'persistent' means in this context? Because I cannot see how network booting leads to the kind of persistence we're talking about. In fact, you seem to be describing technologies that help in implementing the exact opposite. Also, you've missed the point of the Linux comment. I'll spell it out: typically people who revere UNIX in the way described have never in their lives used a UNIX system other than Linux (unless it was MacOSX+, of course). Is that relevant? Not really, that's why it is a footnote.
jl6 5y ago
I get similar benefits from running an application in a VM. If I don’t want to shut down the application, I just save the running state of the VM. But if I need to recover from a crash or other state error, I can still reboot the VM.
ilaksh 5y ago
New operating systems are really needed for virtual and augmented reality. Especially with higher resolution, more comfortable devices that people use for work. I think the 3d file system representation will finally become normal. It will be an exciting time for user interface development. Possibly we could see networked collaboration/"multiplayer" at the OS level.
elteto 5y ago
Not related to Phantom OS, but the OS running the Apollo 11 guidance computers had this same feature. In fact, the famous 1202 program alarms right before landing are related: a low priority task was overwhelming the system and the OS restarted to get a clean slate, continuing the high priority tasks where they left off: “The software rebooted and reinitialized the computer, and then restarted selected programs at a point in their execution flow near where they had been when the restart occurred.” [0] [0] https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html
- garaetjjte 5y ago
  Did it? It seems more like that guidance application was saving checkpoint data, rather than OS feature. Though distinction between OS and applications might be muddy here. In fact persistent OS design would be catastrophic, as restoring all jobs would just cause resource exhaustion again. (more details on Apollo 11 problem: https://www.doneyles.com/LM/Tales.html https://www.doneyles.com/LM/Tales.html)
  - elteto 5y ago
    Well, that's what happened: "On Apollo 11, each time a 1201 or 1202 alarm appeared, the computer rebooted, restarted the important stuff, like steering the descent engine and running the DSKY to let the crew know what was going on, but did not restart all the erroneously-scheduled rendezvous radar jobs." Resource exhaustion was not immediate after reboot because the faulted tasks were low priority and did not get started after reboot right away. I can't even imagine what would have happened if one of the high priority tasks had been the problematic one.
    - garaetjjte 5y ago
      >but did not restart all the erroneously-scheduled rendezvous radar jobs That's wrong. Excess CPU time was stolen by radar counter hardware, not by any software (counters worked by stopping CPU and using its ALU). Problem arised because main guidance routine (SERVICER job) was scheduled always every 2 seconds (by READACCS task/interrupt), and with excess stolen time it didn't finish in time, leading to scheduling of another SERVICER before previous instance finished. This repeated until memory ran out for stacking another SERVICER job. There was no "erroneously-scheduled rendezvous radar jobs", job that needed shedding was multiple stacked instances of SERVICER itself.
strlen 5y ago
Question, how does this compare/contrast to “image based persistence” as used by e.g., most Smalltalk and certain Lisp based VMs?
- webmaven 5y ago
  It is equivalent, as are the pre-XML MS Office file formats which were essentially dumps of the document in memory.
  - strlen 5y ago
    “Every normal man must be tempted, at times, to spit upon his hands, hoist the black flag, and write(2)/WriteFile a struct to disk.”
bigodanktime 5y ago
I'm going to self plug here on work I'm apart of that does persistent processes, although the motivation is different. I do think this paper does a good job of informing the reader on why these sort of features would be really cool that this project does not necessarily dive into. But it requires widening the API and allowing for developers to choose how they persist. https://www.rcs.uwaterloo.ca/pubs/sosp21-aurora.pdf https://www.rcs.uwaterloo.ca/pubs/sosp21-aurora.pdf
dang 5y ago
One past thread. Others? Phantom OS, a Russian OS where “everything is an object” - https://news.ycombinator.com/item?id=19672610 https://news.ycombinator.com/item?id=19672610 - April 2019 (23 comments)
sleepingadmin 5y ago
Isn't the ideal of OS not to need to reboot ever? This project is about intentionally rebooting? I just had the question of 'whatever happened to ksplice?' and immediately found my answer of 'oh oracle' no wonder nobody talks about ksplice anymore.
offmycloud 5y ago
How do you handle backup and restore on a system like this? If the snapshots are at the OS level, and there is cross-app commingling of data objects, how can you restore the state of a particular application or object (formerly "file") without breaking everything? How do I restore a single document or a single contact in my address book?

136 points by maydemir 5y ago

> Its primary goal is to provide environment for programs thatsurvive OS reboot. Such an environment greatly simplifies software development Simplifies? I can't even imagine such a program. To me a program is something that starts and ends and can also re-start with a fresh state in case something goes wrong.

WalterGR 5y ago
What’s the iOS programming model in this regard?
- thatjoeoverthr 5y ago
  Something like “crash-only software”; programs are expected to be killed and relaunched. Software should cope with it, as there is no swap space and ideally the user shouldn’t be managing memory. An app may create to some degree the appearance of persistence but it’s not persistence. Users, also, can easily force kill apps. Ongoing background operations like downloads, uploads and some media playback is delegated to the operating system. Such operations continue when your process is killed, and you check up on them when your process starts. I hope it’s not TMI, but the standard library includes atomic file saves.
  - WalterGR 5y ago
    Thank you!
hsbauauvhabzb 5y ago
What about some middle ground? I’d like my IDE to remain constant between reboots, my browser to some degree is already (if it crashes, it reopens all current tabs on start, which is effectively the same). I would have thought some clever hacks on an existing kernel would be reliable enough rather than an entirely new OS just for one feature, though.
- webmobdev 5y ago
  Isn't that what Hibernation offers on Windows?
  - hsbauauvhabzb 5y ago
    Good point. I’ve not even tried to hibernate a Linux machine in 10 years though so I have no idea if it works there
  - timka 5y ago
    Sort of like that but you don't have to wait until all your 16Gb of RAM are written to disk. AFAU, Dmitry found a cheap way of making consistent snapshots of a running system. BTW, there was a thread on HN relatively recently where someone mentioned that the hard part of hibernation is not restoring the RAM state but bringing all hardware into the right state at boot. That's why hibernation has been so problematic on Linux.
- cdogl 5y ago
  That does sound great, but realistically any application needs to be able to recover from a crash anyway - so even with this support from the OS, you just have a new code path to support. Might be nice for users, but it certainly doesn’t simplify anything for developers.
- jcelerier 5y ago
  The IDE I use, QtCreator, works like this: if you reboot or it crashes you can just restore the previous session on startup and it will bring you exactly where you were.
- qwerty456127 5y ago
  Why do we need this in a kernel? Why not just serialize the essentials of the apps states and re-load it after reboot/sign-in? I'd prefer the desktop environment to remember which apps were running including which documents were opened in them and their windows positions and just re-launch everything at start-up. Given how fast does everything cold-boot today thanks to modern SSDs, I wouldn't even use standby/hibernate if this worked this way.

xupybd 5y ago

I guess turning off and on again won't fix it with this operating system.

kstenerud 5y ago

This kind of system will suffer from the ratchet problem. A single bug that negatively impacts state is no longer fixable by rebooting. Instead, you have to format/reinstall. Unintended state becomes permanent, and upgrade paths constrained to the point where you must do intentional damage. I'd also be leery of fragmentation. The project is falling into the "simplification trap", where all it's doing is moving inherent complexity around (which eventually requires users to do a bunch of work-arounds) rather than eliminating it (because inherent complexity can't be eliminated; only needless complications can be eliminated). In theory, this sort of setup would be nice in a perfect world, but in the real world of buggy software and faulty hardware and cosmic rays and failing network connections, it's a disaster waiting to happen.

stormking 5y ago
Rebooting does not fix any bugs, it just resets the state. I'm not sure this is really the better solution because many bugs never get fixed because "Have you tried turning it off and on, again?"
- kstenerud 5y ago
  It is the better solution because it delineates the difference between important data that must be preserved, and not important data that can be thrown out and regenerated (turning it off and on again). Systems that eliminate this difference massively increase the damage surface of your data by forcing both kinds to be treated equally and even intermix. And since software will always have bugs, your fallout damage increases exponentially. It's a lot like naive state saving code that just dumps an in-memory struct to disk: The moment that struct changes (adding/removing/changing a type or size), your load code breaks.
- thatjoeoverthr 5y ago
  Resetting the state is a necessary step when the state is corrupted. You can’t work around this with “don’t corrupt the state”. So many developers are running the other way: isolate state; reduce dependence on state; eliminate side effects; crash only software; etc. The state doesn’t need to be preserved because it simply isn’t that valuable.

ralferoo 5y ago

I'm not sure this actually solves the real problem that people have. Sure, reboots are annoying and pretty much all current OSes could be improved in this regard. Back in the 90s, it's interesting that SunOS supported live kernel upgrades, so at the same time the kernel on disk was updated, the running system was also patched so it would act the same way as-if rebooted, but without any disruption to the live system. However, in the current days where rebooting is seen as normal for an OS upgrade, but most of the time people just put their computer to sleep, the main problem is that all the network interfaces are effectively useless when the system wakes up because most connections will have long since timed out due to lack of connection acknowledgement packets. In such a case, systems will have to tear-down and re-create a lot of state anyway. The actual issue this seems to solve, that of saving memory state and restoring again, is already solved for most use cases except OS upgrades by using sleep mode. But the hard case about network connections is still unsolved for this system, and by solving the problems it does solve with yet another VM-based environment it'll probably be doomed to obscurity unless common existing applications and virtual machines can easily be made to run on-top of it.

josephg 5y ago
> all the network interfaces are effectively useless when the system wakes up This is - and probably always will be a problem with networked software. You should program as if TCP connections could drop at any time, for any reason. Browser tabs get backgrounded. Laptops go to sleep. Cell phones roam, and go into tunnels. Servers have temporary net splits. Most high level tasks can be retried safely. Nonces and things can be used to safely retry almost anything else. I miss the simplicity of IRC, the way slack and discord smoothly transition between online and offline states is graceful and intuitive. That is how almost all software should behave.
xvilka 5y ago
It's even more essential in IoT and mobile devices, where reboot is often necessary, sometimes just to restore broken connection, like modem. Nobody works now on stable software, everyone tries to ship it as fast as possible. It's not necessarily bad thing, but it is certainly orthogonal to the persistence.
5y ago

Findecanor 5y ago

I wonder how much wear on the system disk is caused unnecessarily by taking continuous snapshots. I think that you would like to limit the actual data to persist to the data that needs to be be persisted — that which can't be recreated quickly, purely from other objects.

mahesh-hegde 5y ago

I remember reading about another, pretty old, microkernel OS which had persistent processes. It was probably a capability based OS. Anyone know what that was?

Findecanor 5y ago
You are probably thinking of EROS <https://en.wikipedia.org/wiki/EROS_(microkernel) https://en.wikipedia.org/wiki/EROS_(microkernel)> But there have been several others also based on similar ideas, such as Mungi. In recent years, there is Twizzler, intended for NVRAM. <https://www.youtube.com/watch?v=0Ix5DYKxzLI https://www.youtube.com/watch?v=0Ix5DYKxzLI>
- mahesh-hegde 5y ago
  Yeah it was EROS.

jcun4128 5y ago

Ooh interesting, probably unrelated but I was thinking a GAI would use one of these at its core eg. "it can't die". For the self/state aspect. Also a criteria is code that modify itself without recompiling. This also considers a power source like a nuclear battery or something that will last a long time or alternate in the low power state/ambient or whatever energy.

t43562 5y ago

I studied some persistent object store databases in the past - which seemed like an incredibly good idea and I think the AS400 used such a database and was very popular. It removed the need for a filesystem or any of the usual patterns for retrieving data like SQL so a whole class of programming that people think of as "normal" today just vaporised. Persistent programs seem like a logical-ish next step but I wish the first step could have been taken because it was very nice to program in.

pjmlp 5y ago

I like the ideas in Phantom, even considering the negative comments, it is refreshing to see people having a go at OS design that isn't just blindly making yet another UNIX clone, because apparently we can't get enough of them.

AnIdiotOnTheNet 5y ago
Indeed. So many people seem to be brought up with an almost religious indoctrination that UNIX[0] is the end-all be-all of OS design. [0] By which of course we mean Linux.
- anthk 5y ago
  - GNU's Not Unix. - Persistent OS'es are a thing since BOOTP boots for example with OpenBSD or alpine on "diskless" mode. Thank Unix for that for being so simple to implement.
  - AnIdiotOnTheNet 5y ago
    Are you sure you know what 'persistent' means in this context? Because I cannot see how network booting leads to the kind of persistence we're talking about. In fact, you seem to be describing technologies that help in implementing the exact opposite. Also, you've missed the point of the Linux comment. I'll spell it out: typically people who revere UNIX in the way described have never in their lives used a UNIX system other than Linux (unless it was MacOSX+, of course). Is that relevant? Not really, that's why it is a footnote.

jl6 5y ago

I get similar benefits from running an application in a VM. If I don’t want to shut down the application, I just save the running state of the VM. But if I need to recover from a crash or other state error, I can still reboot the VM.

ilaksh 5y ago

New operating systems are really needed for virtual and augmented reality. Especially with higher resolution, more comfortable devices that people use for work. I think the 3d file system representation will finally become normal. It will be an exciting time for user interface development. Possibly we could see networked collaboration/"multiplayer" at the OS level.

elteto 5y ago

Not related to Phantom OS, but the OS running the Apollo 11 guidance computers had this same feature. In fact, the famous 1202 program alarms right before landing are related: a low priority task was overwhelming the system and the OS restarted to get a clean slate, continuing the high priority tasks where they left off: “The software rebooted and reinitialized the computer, and then restarted selected programs at a point in their execution flow near where they had been when the restart occurred.” [0] [0] https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html https://www.hq.nasa.gov/alsj/a11/a11.1201-pa.html

garaetjjte 5y ago
Did it? It seems more like that guidance application was saving checkpoint data, rather than OS feature. Though distinction between OS and applications might be muddy here. In fact persistent OS design would be catastrophic, as restoring all jobs would just cause resource exhaustion again. (more details on Apollo 11 problem: https://www.doneyles.com/LM/Tales.html https://www.doneyles.com/LM/Tales.html)
- elteto 5y ago
  Well, that's what happened: "On Apollo 11, each time a 1201 or 1202 alarm appeared, the computer rebooted, restarted the important stuff, like steering the descent engine and running the DSKY to let the crew know what was going on, but did not restart all the erroneously-scheduled rendezvous radar jobs." Resource exhaustion was not immediate after reboot because the faulted tasks were low priority and did not get started after reboot right away. I can't even imagine what would have happened if one of the high priority tasks had been the problematic one.
  - garaetjjte 5y ago
    >but did not restart all the erroneously-scheduled rendezvous radar jobs That's wrong. Excess CPU time was stolen by radar counter hardware, not by any software (counters worked by stopping CPU and using its ALU). Problem arised because main guidance routine (SERVICER job) was scheduled always every 2 seconds (by READACCS task/interrupt), and with excess stolen time it didn't finish in time, leading to scheduling of another SERVICER before previous instance finished. This repeated until memory ran out for stacking another SERVICER job. There was no "erroneously-scheduled rendezvous radar jobs", job that needed shedding was multiple stacked instances of SERVICER itself.

strlen 5y ago

Question, how does this compare/contrast to “image based persistence” as used by e.g., most Smalltalk and certain Lisp based VMs?

webmaven 5y ago
It is equivalent, as are the pre-XML MS Office file formats which were essentially dumps of the document in memory.
- strlen 5y ago
  “Every normal man must be tempted, at times, to spit upon his hands, hoist the black flag, and write(2)/WriteFile a struct to disk.”

bigodanktime 5y ago

I'm going to self plug here on work I'm apart of that does persistent processes, although the motivation is different. I do think this paper does a good job of informing the reader on why these sort of features would be really cool that this project does not necessarily dive into. But it requires widening the API and allowing for developers to choose how they persist. https://www.rcs.uwaterloo.ca/pubs/sosp21-aurora.pdf https://www.rcs.uwaterloo.ca/pubs/sosp21-aurora.pdf

dang 5y ago

One past thread. Others? Phantom OS, a Russian OS where “everything is an object” - https://news.ycombinator.com/item?id=19672610 https://news.ycombinator.com/item?id=19672610 - April 2019 (23 comments)

sleepingadmin 5y ago

Isn't the ideal of OS not to need to reboot ever? This project is about intentionally rebooting? I just had the question of 'whatever happened to ksplice?' and immediately found my answer of 'oh oracle' no wonder nobody talks about ksplice anymore.

offmycloud 5y ago

How do you handle backup and restore on a system like this? If the snapshots are at the OS level, and there is cross-app commingling of data objects, how can you restore the state of a particular application or object (formerly "file") without breaking everything? How do I restore a single document or a single contact in my address book?