5 ms·

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case. But CoW index write amplification you say? Well now let's talk about infrastructure differences. Datomic can use s3 as a storage layer. Cold data storage is then $10/month/terabyte. And compaction is trivially non-blocking in a CoW dataset. It can be done by EC2 spec instances whenver is convenient with zero impact on the production system. Don't generalize the experiences of a few folks running couchdb on a couple in house servers to people working with a fundamentally different approach on fundamentally different infrastructure. Back to the cheap cost of cold storage, for most businesses that's well into "who cares" cost territory, but having the data also allows them to preserve full lineage in the advent they discover a data loss bug in their application. That could potentially be quite valuable. One way to think of these systems is that they have integrated backup data. MySQL folks generally don't count the size of backup sets as part of the database footprint, and common backup schedules (eg father, grandfather) provide only a limited ability to dig back into the past. By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently? The answer is largely one of implementation constraints: text is cheap, vcs doesn't index the way databases do, etc. But as storage and processing densities continue to grow more datasets shift to the trivial category. As much as many HNews posters dream of being in a 'big data' shop, the vast majority of startups have maybe a couple GB in MySQL or Postgres. Our status quo toolset and approach is discarding value for these folks. Maybe you think that's fine, but some of us are going to keep pushing to enable new capabilities and advantages. Increasingly these implementation constraints will disappear as people attack them. It'd be nice to not suffer scornful surface criticisms from people who base their career around the status quo.

jeffdavis 13y ago
I think you are making a mistake by reading it as a critique of immutability in general. Rather, I think the author is criticizing an overly-simplisitic approach to using immutability. When he describes traditional databases: "What’s happening in such a database is a combination of short-term immutability, read and write optimizations to save and/or coalesce redundant work, and continuous “compaction” and reuse of disk space to stabilize disk usage and avoid infinite growth." He's saying that's what people arrive at after working out the issues with immutability. When you say: "Increasingly these implementation constraints will disappear as people attack them." the author would probably respond: "Yes, and people will attack them with a lot of the same strategies that SQL databases are already using today".
- jasonwatkinspdx 13y ago
  I strongly disagree with the idea that temporal databases will be forced to revise themselves to the implementation designs of Postgres, MySQL, etc. In fact, I'd make the firm prediction that the paged window CoW strategy of LMDB will become dominant for these more traditional database designs, and that a new class of temporal databases will improve upon that architecture.
  - jeffdavis 13y ago
    It's not so much that everything will look exactly like postgres. It's that the biggest difference right now between the designs is complexity (the kind that comes with maturity). Every mature general-purpose database system makes use of both CoW/immutable design principles as well as locking and update-in-place in various contexts. It remains to be seen how "temporal databases" (as you call them) will adapt to real-world demands, but it might not look as radically different as you think. Also, for anyone who thinks that CoW is the be-all-end-all design principle, I have my doubts. The main reason is that it doesn't compose very cleanly -- a design that involves CoW over CoW over CoW starts to look wasteful after a while. That being said, I would not be surprised if some of these new databases found some new trade-offs that worked very nicely and got adopted by more traditional systems. Storage is not nearly as much of a concern for many kinds of problems, and it seems natural that such a shift would lead to some design changes.
homerowilson 13y ago
"By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently?" Well said!
- jeltz 13y ago
  Git is not append only though. It has a stop the world GC which compacts data and removes objects not referenced by any current branch, tag or similar object. So even if you want to keep the history you may not want to keep everything.
  - jasonwatkinspdx 13y ago
    It's important to make a distinction between semantic and physical datasets. Git is append only (or more properly, strictly accumulative in an information sense) from a semantic perspective. From a user perspective we don't really care what the packfiles etc look like. Git also will forget timelines (transitive closure of a reference) you don't care about if you tell it to. So it's important to distinguish purely persistent data structures in the sense of Okasaki et all, and append only datasets in the semantic sense. Git still qualifies as the latter even though it's implementation uses mutability as appropriate. This is a good example of where the OP's critique is painting with too broad a brush. If we think from first principles: indexes are derived state, not primary state. The value log is the ultimate authority, the index can be regenerated at will. There is a huge design space of indexing over append only logs that remains largely unexplored. I personally strongly believe databases will move emphatically in this direction because the benefits are so great. Given the typical tiered web application, wouldn't you like it if when you get an exception notification instead of just a text blob of a stack trace you can resume a continuation of the state of the system as the user saw it at the time of the error? Navigate forward and backward in time at will, even with nonlinear jumps? Edit the source code of the past and then replay toward the future to see if it resolves or reproduces the exception? There are huge opportunities here. I will admit it's not immediately clear how to implement a database that reflects these ideas. It's going to take some experimentation and learning. That'll involve some failed experiments. What bugged me most about the OP was the attitude of "we're done, shut up with this other stuff." We are emphatically not done. It's ironic to note that at the moment MySQL was birthed it faced the same FUD style criticism.
jasonwatkinspdx 13y ago
s/analytically/analytic/
corresation 13y ago
By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently? All mainstream database products allow you to retain the entire history of every change ever made in the form of transaction logs. With those logs you can recover your database to any specific point in time, and to unwind specific transactions. Of course most discard transaction logs because the volumes tend to be huge in many mainstream, operating databases. I worked on one system where the main database barely pushed 10GB, but there were 100s of GBs of transaction logs generated daily. However the core reason that mainstream databases discard with historical data is performance, not size -- if your current users overwhelmingly only care about the current state of the data, and it is rare that you need to go back into the past (which is what transaction logs and snapshots provide), paying an extremely high performance penalty to retain all of that historical data does not pay off. And there are no immutable products that offer similar performance to mainstream products under most usage scenarios, instead forcing you into extremely confined uses. It'd be nice to not suffer scornful surface criticisms from people who base their career around the status quo. This is a garbage non sequitur, as an aside. It is a disgraceful attempt -- as so commonly happens in such discussion -- to try to attach agenda to an opinion that one doesn't like.
- nickik 13y ago
  Well, I have worked in a few companys with big SQL databases. You say the only care about the current state, but inreally EVER big database I have seen actually impnents some kind of backup thingy. Mostly handcrafted stuff. The diffrence between Datomic and just storing your transaction log is that datomic gives you first class access to your hole history, you can work with it as easly as with the current data. I have at least not seen that in any other database. So for me at least it seams a database who just stores everything and gives access to it is what most people should use for most problems. For me at least Datomic is the new standard and only if I have a special use case I would go to something else.
  - corresation 13y ago
    you can work with it as easily as with the current data. I have at least not seen that in any other database. At the cost that working with your current data is as hard as working with the historical data. This does not come for free, and the only way to have such a versioned history is at a significant cost to generalized query performance -- sure certain things (like a single scalar lookup) can be fast, but generalized queries will be terrible. I'm sure there will be some map/reduce claimed solution. For me at least Datomic is the new standard and only if I have a special use case I would go to something else. This is an incredible and rather ridiculous statement.

jeffdavis 13y ago

I think you are making a mistake by reading it as a critique of immutability in general. Rather, I think the author is criticizing an overly-simplisitic approach to using immutability. When he describes traditional databases: "What’s happening in such a database is a combination of short-term immutability, read and write optimizations to save and/or coalesce redundant work, and continuous “compaction” and reuse of disk space to stabilize disk usage and avoid infinite growth." He's saying that's what people arrive at after working out the issues with immutability. When you say: "Increasingly these implementation constraints will disappear as people attack them." the author would probably respond: "Yes, and people will attack them with a lot of the same strategies that SQL databases are already using today".

jasonwatkinspdx 13y ago
I strongly disagree with the idea that temporal databases will be forced to revise themselves to the implementation designs of Postgres, MySQL, etc. In fact, I'd make the firm prediction that the paged window CoW strategy of LMDB will become dominant for these more traditional database designs, and that a new class of temporal databases will improve upon that architecture.
- jeffdavis 13y ago
  It's not so much that everything will look exactly like postgres. It's that the biggest difference right now between the designs is complexity (the kind that comes with maturity). Every mature general-purpose database system makes use of both CoW/immutable design principles as well as locking and update-in-place in various contexts. It remains to be seen how "temporal databases" (as you call them) will adapt to real-world demands, but it might not look as radically different as you think. Also, for anyone who thinks that CoW is the be-all-end-all design principle, I have my doubts. The main reason is that it doesn't compose very cleanly -- a design that involves CoW over CoW over CoW starts to look wasteful after a while. That being said, I would not be surprised if some of these new databases found some new trade-offs that worked very nicely and got adopted by more traditional systems. Storage is not nearly as much of a concern for many kinds of problems, and it seems natural that such a shift would lead to some design changes.

homerowilson 13y ago

"By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently?" Well said!

jeltz 13y ago
Git is not append only though. It has a stop the world GC which compacts data and removes objects not referenced by any current branch, tag or similar object. So even if you want to keep the history you may not want to keep everything.
- jasonwatkinspdx 13y ago
  It's important to make a distinction between semantic and physical datasets. Git is append only (or more properly, strictly accumulative in an information sense) from a semantic perspective. From a user perspective we don't really care what the packfiles etc look like. Git also will forget timelines (transitive closure of a reference) you don't care about if you tell it to. So it's important to distinguish purely persistent data structures in the sense of Okasaki et all, and append only datasets in the semantic sense. Git still qualifies as the latter even though it's implementation uses mutability as appropriate. This is a good example of where the OP's critique is painting with too broad a brush. If we think from first principles: indexes are derived state, not primary state. The value log is the ultimate authority, the index can be regenerated at will. There is a huge design space of indexing over append only logs that remains largely unexplored. I personally strongly believe databases will move emphatically in this direction because the benefits are so great. Given the typical tiered web application, wouldn't you like it if when you get an exception notification instead of just a text blob of a stack trace you can resume a continuation of the state of the system as the user saw it at the time of the error? Navigate forward and backward in time at will, even with nonlinear jumps? Edit the source code of the past and then replay toward the future to see if it resolves or reproduces the exception? There are huge opportunities here. I will admit it's not immediately clear how to implement a database that reflects these ideas. It's going to take some experimentation and learning. That'll involve some failed experiments. What bugged me most about the OP was the attitude of "we're done, shut up with this other stuff." We are emphatically not done. It's ironic to note that at the moment MySQL was birthed it faced the same FUD style criticism.

jasonwatkinspdx 13y ago

s/analytically/analytic/

corresation 13y ago

By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently? All mainstream database products allow you to retain the entire history of every change ever made in the form of transaction logs. With those logs you can recover your database to any specific point in time, and to unwind specific transactions. Of course most discard transaction logs because the volumes tend to be huge in many mainstream, operating databases. I worked on one system where the main database barely pushed 10GB, but there were 100s of GBs of transaction logs generated daily. However the core reason that mainstream databases discard with historical data is performance, not size -- if your current users overwhelmingly only care about the current state of the data, and it is rare that you need to go back into the past (which is what transaction logs and snapshots provide), paying an extremely high performance penalty to retain all of that historical data does not pay off. And there are no immutable products that offer similar performance to mainstream products under most usage scenarios, instead forcing you into extremely confined uses. It'd be nice to not suffer scornful surface criticisms from people who base their career around the status quo. This is a garbage non sequitur, as an aside. It is a disgraceful attempt -- as so commonly happens in such discussion -- to try to attach agenda to an opinion that one doesn't like.

nickik 13y ago
Well, I have worked in a few companys with big SQL databases. You say the only care about the current state, but inreally EVER big database I have seen actually impnents some kind of backup thingy. Mostly handcrafted stuff. The diffrence between Datomic and just storing your transaction log is that datomic gives you first class access to your hole history, you can work with it as easly as with the current data. I have at least not seen that in any other database. So for me at least it seams a database who just stores everything and gives access to it is what most people should use for most problems. For me at least Datomic is the new standard and only if I have a special use case I would go to something else.
- corresation 13y ago
  you can work with it as easily as with the current data. I have at least not seen that in any other database. At the cost that working with your current data is as hard as working with the historical data. This does not come for free, and the only way to have such a versioned history is at a significant cost to generalized query performance -- sure certain things (like a single scalar lookup) can be fast, but generalized queries will be terrible. I'm sure there will be some map/reduce claimed solution. For me at least Datomic is the new standard and only if I have a special use case I would go to something else. This is an incredible and rather ridiculous statement.