3 ms·

SQL remains a useful tool in the box, but while SQL and NoSQL are fighting it out to be the default go-to database technology, something else on the horizon is

SQL remains a useful tool in the box, but while SQL and NoSQL are fighting it out to be the default go-to database technology, something else on the horizon is threatening the concept of having any database separate from the app at all: persistent in-application state. We don't need a database if we can just load the application state in RAM and save it back via serialization. This works well in surprisingly many cases.

nawitus 12y ago
The problems with that are easy to see though: you need lots of RAM if your database is non-trivial in size. You also need to write the state to the database continuously to prevent data loss from failures. Not saying the idea doesn't have use cases, of course. Redis (for example) is pretty popular these days, but I don't see how it'll become the default go-to database.
- mantrax5 12y ago
  Yes, you need to write continuously. Basically you need a transaction log. Redis unfortunately botches that part a bit (it has no performant and reliable way of persisting every change before acknowledging a "commit"), but that's roughly the idea. However not all of your state will be persistent, so you write only the persistent parts. For example, if you need to have an index for quick access to items based on some criteria, you don't need to store the index on disk, you can always rebuild it on startup from the rest of the data. While there's an upper limit for the amount of data you can store this way (it has to fit in RAM), it fits very favorably in web scenarios, where you have read-heavy write-light data access patterns, and you never have to touch the disk in order to read a piece of data (guaranteed).
_delirium 12y ago
That's a fairly traditional no-DB approach [1], but what's the reason to believe the dominance of that approach is on the horizon? [1] Common Lisp has a particularly nice system for doing it transparently, http://common-lisp.net/project/elephant/ http://common-lisp.net/project/elephant/, and I believe HN is implemented using a DB-via-serialization strategy as well
- mantrax5 12y ago
  Actor systems. Actor systems allow this to scale and cross the machine boundary without having some dedicated database product doing it for us. I've noticed that the more I work with actor systems (each actor is stateful and keeps its own state), the less I use databases. In fact, the idea of keep state separate from the code reading it and changing it begins to look oddly broken as a concept.

nawitus 12y ago

The problems with that are easy to see though: you need lots of RAM if your database is non-trivial in size. You also need to write the state to the database continuously to prevent data loss from failures. Not saying the idea doesn't have use cases, of course. Redis (for example) is pretty popular these days, but I don't see how it'll become the default go-to database.

mantrax5 12y ago
Yes, you need to write continuously. Basically you need a transaction log. Redis unfortunately botches that part a bit (it has no performant and reliable way of persisting every change before acknowledging a "commit"), but that's roughly the idea. However not all of your state will be persistent, so you write only the persistent parts. For example, if you need to have an index for quick access to items based on some criteria, you don't need to store the index on disk, you can always rebuild it on startup from the rest of the data. While there's an upper limit for the amount of data you can store this way (it has to fit in RAM), it fits very favorably in web scenarios, where you have read-heavy write-light data access patterns, and you never have to touch the disk in order to read a piece of data (guaranteed).

_delirium 12y ago

That's a fairly traditional no-DB approach [1], but what's the reason to believe the dominance of that approach is on the horizon? [1] Common Lisp has a particularly nice system for doing it transparently, http://common-lisp.net/project/elephant/ http://common-lisp.net/project/elephant/, and I believe HN is implemented using a DB-via-serialization strategy as well

mantrax5 12y ago
Actor systems. Actor systems allow this to scale and cross the machine boundary without having some dedicated database product doing it for us. I've noticed that the more I work with actor systems (each actor is stateful and keeps its own state), the less I use databases. In fact, the idea of keep state separate from the code reading it and changing it begins to look oddly broken as a concept.