3 ms·
Transactions (something like fsync) and code upgrades (something like database migrations or hot-reload in a debugger) seem like the tricky parts and I don't se
by skybrian 5y ago
Transactions (something like fsync) and code upgrades (something like database migrations or hot-reload in a debugger) seem like the tricky parts and I don't see a discussion of how they'll handle it.
Let's assume you get a stream of security patches from somewhere. How do you apply them without a reboot? What if a data structure needs to be migrated?
During early development, it's easy to just throw away all your data and start over after a redesign, but as with production databases or filesystem implementations, once you're storing real user data, you're not allowed to do that anymore. It's helpful architecturally to have a distinction between on-disk structures (which you're careful about either not changing or migrating) and in-memory structures (which don't) rather than trying to freeze or migrate everything.
- deleted 5y ago[deleted]
- zozbot234 5y agoTransactions can be done "easily" by reusing virtual memory support to trap accesses to memory (read/write/execute) and basically running coherence protocols in software. This is also how distributed transactions can be implemented, which enable you to deploy a bunch of threads running in a single address space across separate nodes in a cluster. It's not all that much harder to implement than checkpoint or restore, which they seem to have already.
- bitcharmer 5y agoIf I understand this idea correctly, it would have a huge performance impact if every load and store had to be trapped. Or am I missing something here?
- zozbot234 5y agoNot every access, but those that have to be trapped in order to maintain coherence. The cost of a trap might simply be in the same ballpark as other kinds of synchronization.
- vanviegen 5y ago> It's helpful architecturally to have a distinction between on-disk structures (which you're careful about either not changing or migrating) and in-memory structures (which don't) rather than trying to freeze or migrate everything. That's true, but syncing between the database and in memory data structures is tedious. Imagine building an application and never having to worry about storage. Just instantiate and reference objects and you're set. When working with lots of data, the OS would just swap things in on demand. That would save a lot of development time for many types of applications. But yes, there would be some big challenges regarding application updates and how not to destroy all data with a trivial bug. (A misplaced `customers = []` might be all it takes to permanently wipe all data. Perhaps this idea should be combined with immutable data structures and versioning.) [Edit] After some more consideration: what I'm talking about feels more like a language feature than something the operating system can/should do.
- skybrian 5y agoSerialization doesn't go away because you still need it for network connections. Data formats useful for sending messages over the network can also be useful for saving things to disk. These formats should be OS and language-independent and have rules for backward compatibility. Consider how hard it would be to replicate a git repository if it's just a bunch of ad-hoc classes defined in one language. Protobufs are one design that addresses these concerns. There are others. There is an impedence mismatch, but maybe this would be better addressed through languages that have better support for serialization?
- phoe-krk 5y agoFor code upgrades, see e.g. multithreaded Common Lisp implementations which achieve this consistently via extremely late binding. Of course, things get a bit more complex if you e.g. need to atomically synchronize upgrading multiple code bodies rather than only one, but that is something that can be done in application logic itself.