4 ms·
Well here's one good reason: If your data cannot all fit in memory at once, you cannot simply serialize and deserialize it from disk. You will have to create so
by astine 13y ago
Well here's one good reason: If your data cannot all fit in memory at once, you cannot simply serialize and deserialize it from disk. You will have to create some kind of indexed solution involving reading and writing to files piecemeal and by doing so you'll have created an impromptu database by default.
Other problems including loosing data if your application crashes and isn't able save it's data, performance on very large datasets, lack of concurrent access, etc
Generally, flat files are fine for low availability applications with small datasets. As your application grows however, you generally need to find more robust methods of persisting your application. Picking a persistence engine/database which matches you application will be essential at some point. It need not be a RDBMs, as 'NoSQL' databases and object databases may fit your application better.
One more thought: A lot of the fuss about 'NoSQL' vs RDBMs come from people trying to treat RDBMs strictly as persistence engines. That is, they start with an application that run well in memory and then realize that they need to deal with one or more of the issues I raised above. They pull out a RDBM and are annoyed that they now have map the data from the perfectly fine datastructures they were already using to a normalized tabular structure.
The reason this happens is because RDBMs were created to solve a different class of problems than simple persistence. They are for when you are starting out with data that needs to be housed and organized, as opposed to an application that simply needs to save its data.
- lsb 13y agoNo, you'll have created a paging system, and it'll probably fight with your OS's pager. The disk paging system is but one of the components of a database, but what makes a database a database is the leverage that you get over your data, versus just a blob on disk.
- Arwill 13y agoActually what i find to be the core question in deciding to use a database or not, is whether all data can be loaded at program start into memory or not. Loading all data at start is pretty simple and efficient. In this regards, there is another problem, getting the changes from memory back to the disk. Writing all data to disk, and even overwriting existing data is out of question, so such a system is needed which saves the changed data incrementally. Is there such a "database", which: a. loads up all the data at the start + loads the incremental changes, b. writes the incremental changes in transactional manner to disk, while not messing up the OS paging? I wonder if there is such a "persistent paging file" library available?
- marcosdumay 13y ago>Is there such a "database", which:...? As far as I know, no, all of them are lazy and load only what you query into the memory. But if your data fits in memory, it'll stay there, and subsequent queries will be faster. Also, you can take the quotation marks from database, you have some quite generic requisites, most of them implement all requisits, except for the caching at startup one.
- delian66 13y agoI think you may be interested in checking out Redis (it is an in-memory data structure server); see especially http://redis.io/topics/persistence#append-only-file http://redis.io/topics/persistence#append-only-file .