4 ms·

"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal dis

"By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption." The cure is surely worse than the disease! The program state in memory at the time of abnormal termination is likely to be inconsistent, leading to an unusable file. The subset of that that happens to have been committed to disk is likely to be worse. Contrast with the traditional approach of explicit serialisation / deserialisation for persistence: * serialisation occurs when the program is in a known state. We can reason about what invariants hold at the point when state is sent to storage. We can take steps to confirm that data actually has been committed to persistent storage before we treat the operation as complete. * deserialisation recreates program state in a clean environment from minimal data. This has the side effect of reinitialising state that may have caused the unexpected termination. Coupled with the ability to reason about program state at the point of serialisation it makes it feasible to attempt error recovery. * Explicit serialisation and deserialisation stages make it possible to construct persistent state that is portable across environments and through development mmap() is blindingly fast, but it is a tradeoff between performance, complexity and reliability - naive use across the board is unlikely to improve all three!

ninkendo 13y ago
Also, beyond only the state in the file being inconsistent, even if you manage to get a complete atomic write of your program state before you crash, the very same program state that caused the crash (if the crash was your own program's fault) is now being faithfully read back from disk! You're just asking to crash again immediately by blindly loading it all back.
- marvy 13y ago
  Indeed. Just like this: http://thedailywtf.com/Articles/Serial-Port-Pit-Bull.aspx http://thedailywtf.com/Articles/Serial-Port-Pit-Bull.aspx
- hyc_symas 12y ago
  This is why you must have ACID transactions to use this approach. Even without mmap, if your program successfully writes an invalid data structure to disk, the next time you run and read it back, you will crash. That's orthogonal to the method used to write and read the program state.
hyc_symas 12y ago
Portability of binary data structures across environments is IMO irrelevant. We export to plaintext for migrations. Writing in a binary format that is portable across both 32 and 64 bit CPU architectures will be a waste on one or the other of the archs, and the point of approaches like this is to optimize runtime performance. Your point about serialisation occurring with a known state is not lost in an mmap approach. If you have transactional writes (as LMDB does), then writing to the mmap also only occurs with a known state. In reality, e.g. in OpenLDAP, you wind up with a minimal serialisation step, but with zero deserialisation. E.g., if I want to store a record: typedef struct foo { int len; char *data; } foo; The struct and the data of the char ptr may originally have been obtained from two separate mallocs and thus reside at very different locations in memory. My serializer would simply allocate a new block and store the character data contiguously with the end of the struct. If you mmap at a fixed address, then you can store the address of the character data directly in the struct's "data" field, and the next time you come around to use this record, no deserialization is needed. Explicit deserialization is truly just a waste.

ninkendo 13y ago

Also, beyond only the state in the file being inconsistent, even if you manage to get a complete atomic write of your program state before you crash, the very same program state that caused the crash (if the crash was your own program's fault) is now being faithfully read back from disk! You're just asking to crash again immediately by blindly loading it all back.

marvy 13y ago
Indeed. Just like this: http://thedailywtf.com/Articles/Serial-Port-Pit-Bull.aspx http://thedailywtf.com/Articles/Serial-Port-Pit-Bull.aspx
hyc_symas 12y ago
This is why you must have ACID transactions to use this approach. Even without mmap, if your program successfully writes an invalid data structure to disk, the next time you run and read it back, you will crash. That's orthogonal to the method used to write and read the program state.

hyc_symas 12y ago

Portability of binary data structures across environments is IMO irrelevant. We export to plaintext for migrations. Writing in a binary format that is portable across both 32 and 64 bit CPU architectures will be a waste on one or the other of the archs, and the point of approaches like this is to optimize runtime performance. Your point about serialisation occurring with a known state is not lost in an mmap approach. If you have transactional writes (as LMDB does), then writing to the mmap also only occurs with a known state. In reality, e.g. in OpenLDAP, you wind up with a minimal serialisation step, but with zero deserialisation. E.g., if I want to store a record: typedef struct foo { int len; char *data; } foo; The struct and the data of the char ptr may originally have been obtained from two separate mallocs and thus reside at very different locations in memory. My serializer would simply allocate a new block and store the character data contiguously with the end of the struct. If you mmap at a fixed address, then you can store the address of the character data directly in the struct's "data" field, and the next time you come around to use this record, no deserialization is needed. Explicit deserialization is truly just a waste.