11 ms·

In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of

In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of an application. Data may also be more valuable to the organization than the code that processes it. While I'm no fan of databases, one obvious advantage is that they provide direct access to the data in a standard way that is decoupled from the specific application code. This makes it easy to perform migrations, backups etc. It also increases one's confidence in the data integrity. Any solution that aims to replace databases altogether must address these concerns. I think that intimately coupling data with the application state, as suggested in the article, does not achieve this.

alexro 15y ago
And not the last thing is security and permissions to access different parts of data. I see no way to have it easily implemented in the event logging system.
- mbreese 15y ago
  Chmod? Seriously, if you need different permissions to stres in an event log, just write multiple event logs - each with only the data they need - and store them with different permissions. This assumes that they can be properly decoupled, but since you're the one writing the event log, you can set it up however you want.
  - alexro 15y ago
    Yeah, right. Adding more pieces to the puzzle makes it more interesting to solve!
wpietri 15y ago
The goal is not to replace databases altogether. The goal is to solve some particular problems very well. Last time I used this approach, for example, we mirrored a bunch of data in a traditional SQL store for reporting and ad-hoc querying, things that databases are great at. In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible. I also think data integrity is easier to maintain with a system like this. SQL constraints don't allow me to express nearly as much about data integrity as I can in code. Sure, I could use stored procedures, but if I'm going to write code somewhere, I'd rather it be in my app.
- huherto 15y ago
  "With multiple code bases touching the same data, schema improvements become nearly impossible." Few companies have procedures in place that allow this; but it is possible if you have the right procedures.
  - wpietri 15y ago
    Sure, but the problem becomes harder the larger you get. Look at almost any Internet-wide deployment, though, and you see the alternative: isolate database schemas behind APIs, and rev APIs and schemas separately, as the situation demands.
    - kfool 15y ago
      Isolating databases behind APIs and rev-ing API+schema separately is not enough. When the schema changes, data must be transformed to match the new schema version. As you point out this takes too long with a large database, and it doesn't account for data consistency. We have been working on building what we hope are the procedures for this with ChronicDB (http://chronincdb.com http://chronincdb.com). But it turned out harder than it seems, and we are not sure it will quite work out. We'd welcome feedback.
- jeffdavis 15y ago
  "I also think data integrity is easier to maintain with a system like this." If you are in the middle of a transaction and you realize that some constraint is being violated, how do you roll it back without interfering with the other transactions?
  - wpietri 15y ago
    I can't speak to all systems like this, but the Prevayler approach is pretty straightforward. Most importantly, there are no simultaneous transactions: changes happen one at a time. That seems crazy if you're used to dealing with disk-backed databases, but if everything is hot in RAM, then it's not a problem. In that context, it's pretty easy: when you start executing a change you verify all your constraints before doing anything.
jcromartie 15y ago
When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, by creating a SQL database for other applications to query).
- alexro 15y ago
  You can have a design where your previous version of a record gets automatically copied into another table along with the timestamp of the operation. Then you can slice this history however you want. All with no additional app code. But I wouldn't write off the noDB approach for various transitional data, or data that isn't mean to live long anyway, like tweets.
  - enjo 15y ago
    Why would the lifetime of the data be at all relevant? Most noDB (a term I don't like much) stores are built to be highly durable. There are a lot of questions that make the choice of data store a difficult one, but I'm not sure that plays into it at all.
- matwood 15y ago
  When people set out to design a data driven application, they usually end up updating and deleting records. FTFY... It's not hard to build history into a SQL table design. You can even store events in a...wait for it... SQL database. I have built numerous systems backed by SQL databases that have complete history retention. Answering questions like 'who had id 'X' on this date 3 years ago' are easily solvable with basic standard sql. I certainly don't believe SQL databases are perfect or the tool for every job, but in many cases they work just fine until you get into very large datasets. Admittedly, I only deal with databases in the 100s of GB range so I have yet to personally run into the scaling problems that a Google or Facebook have and the SQL backed systems I have built work just fine.

alexro 15y ago

And not the last thing is security and permissions to access different parts of data. I see no way to have it easily implemented in the event logging system.

mbreese 15y ago
Chmod? Seriously, if you need different permissions to stres in an event log, just write multiple event logs - each with only the data they need - and store them with different permissions. This assumes that they can be properly decoupled, but since you're the one writing the event log, you can set it up however you want.
- alexro 15y ago
  Yeah, right. Adding more pieces to the puzzle makes it more interesting to solve!

wpietri 15y ago

The goal is not to replace databases altogether. The goal is to solve some particular problems very well. Last time I used this approach, for example, we mirrored a bunch of data in a traditional SQL store for reporting and ad-hoc querying, things that databases are great at. In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible. I also think data integrity is easier to maintain with a system like this. SQL constraints don't allow me to express nearly as much about data integrity as I can in code. Sure, I could use stored procedures, but if I'm going to write code somewhere, I'd rather it be in my app.

huherto 15y ago
"With multiple code bases touching the same data, schema improvements become nearly impossible." Few companies have procedures in place that allow this; but it is possible if you have the right procedures.
- wpietri 15y ago
  Sure, but the problem becomes harder the larger you get. Look at almost any Internet-wide deployment, though, and you see the alternative: isolate database schemas behind APIs, and rev APIs and schemas separately, as the situation demands.
  - kfool 15y ago
    Isolating databases behind APIs and rev-ing API+schema separately is not enough. When the schema changes, data must be transformed to match the new schema version. As you point out this takes too long with a large database, and it doesn't account for data consistency. We have been working on building what we hope are the procedures for this with ChronicDB (http://chronincdb.com http://chronincdb.com). But it turned out harder than it seems, and we are not sure it will quite work out. We'd welcome feedback.
jeffdavis 15y ago
"I also think data integrity is easier to maintain with a system like this." If you are in the middle of a transaction and you realize that some constraint is being violated, how do you roll it back without interfering with the other transactions?
- wpietri 15y ago
  I can't speak to all systems like this, but the Prevayler approach is pretty straightforward. Most importantly, there are no simultaneous transactions: changes happen one at a time. That seems crazy if you're used to dealing with disk-backed databases, but if everything is hot in RAM, then it's not a problem. In that context, it's pretty easy: when you start executing a change you verify all your constraints before doing anything.

jcromartie 15y ago

When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, by creating a SQL database for other applications to query).

alexro 15y ago
You can have a design where your previous version of a record gets automatically copied into another table along with the timestamp of the operation. Then you can slice this history however you want. All with no additional app code. But I wouldn't write off the noDB approach for various transitional data, or data that isn't mean to live long anyway, like tweets.
- enjo 15y ago
  Why would the lifetime of the data be at all relevant? Most noDB (a term I don't like much) stores are built to be highly durable. There are a lot of questions that make the choice of data store a difficult one, but I'm not sure that plays into it at all.
matwood 15y ago
When people set out to design a data driven application, they usually end up updating and deleting records. FTFY... It's not hard to build history into a SQL table design. You can even store events in a...wait for it... SQL database. I have built numerous systems backed by SQL databases that have complete history retention. Answering questions like 'who had id 'X' on this date 3 years ago' are easily solvable with basic standard sql. I certainly don't believe SQL databases are perfect or the tool for every job, but in many cases they work just fine until you get into very large datasets. Admittedly, I only deal with databases in the 100s of GB range so I have yet to personally run into the scaling problems that a Google or Facebook have and the SQL backed systems I have built work just fine.