3 ms·

> One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of de

> One very key architectural decision for a system is the degree of reliability that it should possess. This is an explicit trade-off between the rapidity of development (in particular the level of indoctrination needed before new contributors are able to augment the feature set) and the operational stability. By preserving state explicitly to memory backed files, several classes of unexpected events causing the program to crash can be recovered from with minimal disruption. The benefit here is that the system can be developed rapidly with code reviews focusing on data integrity and paying less attention to complex interactions that can lead to crashes. The mmapped files are going to break every time you make the smallest change to your data structures. This strikes me as not terribly useful for rapid development.

Negitivefrags 13y ago
That depends on how they were created. On our game project we use a data tool that allows you to create tables of data out of columns with specific data types as well as references to other table rows. When you export from the tool it creates data files but also generates the C++ struct that represents the schema. The loader simply mmaps the data in to memory. Variable length data like strings gets packed at the end of the table with the fixed length records containing relative pointers in to it. The data on disk is stored in the exact format that is needed at run time. References to rows in other tables turn in to pointer like objects so you get a perfectly natural syntax as if the data had been loaded in to hand written classes. It's perfect for rapid development because designers can add new columns or tables of data and then just tell the programmer what they are called. It also loads lightning fast and has excellent run time properties due to cache coherency.
mtdewcmu 13y ago
This sounds like it could be a good fit for mobile on iOS (I can't speak to android). If the app has significant startup costs, this lets you basically save a core dump of those structures that were slow to build. Then you could have an alternate fast startup mode that reads from a core dump.
icedchai 13y ago
You can plan ahead and leave "spare" space in the memory mapped structures for future expansion. Combined with versioning, you can take this quite far. I've worked on financial transaction processing systems using memory mapped files as their primary means of data storage. It is very effective.
hyc_symas 12y ago
Your data structure design needs serious help if this is a problem for you. In OpenLDAP we store LDAP entries which can contain variable numbers of attributes and variable numbers of values per attribute, all in a unified format. Redefining your data structures is as simple as adding or removing an attribute or value from an entry. No sweat. Rolling your own data model is probably a bad idea...

Negitivefrags 13y ago

That depends on how they were created. On our game project we use a data tool that allows you to create tables of data out of columns with specific data types as well as references to other table rows. When you export from the tool it creates data files but also generates the C++ struct that represents the schema. The loader simply mmaps the data in to memory. Variable length data like strings gets packed at the end of the table with the fixed length records containing relative pointers in to it. The data on disk is stored in the exact format that is needed at run time. References to rows in other tables turn in to pointer like objects so you get a perfectly natural syntax as if the data had been loaded in to hand written classes. It's perfect for rapid development because designers can add new columns or tables of data and then just tell the programmer what they are called. It also loads lightning fast and has excellent run time properties due to cache coherency.

mtdewcmu 13y ago

This sounds like it could be a good fit for mobile on iOS (I can't speak to android). If the app has significant startup costs, this lets you basically save a core dump of those structures that were slow to build. Then you could have an alternate fast startup mode that reads from a core dump.

icedchai 13y ago

You can plan ahead and leave "spare" space in the memory mapped structures for future expansion. Combined with versioning, you can take this quite far. I've worked on financial transaction processing systems using memory mapped files as their primary means of data storage. It is very effective.

hyc_symas 12y ago

Your data structure design needs serious help if this is a problem for you. In OpenLDAP we store LDAP entries which can contain variable numbers of attributes and variable numbers of values per attribute, all in a unified format. Redefining your data structures is as simple as adding or removing an attribute or value from an entry. No sweat. Rolling your own data model is probably a bad idea...