5 ms·

Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design? Honest question, I'm curious.

by Toshio 13y ago

zimbatm 13y ago
I see the created_at and updated_at fields as the poor man's datomic.
YuriNiyazov 13y ago
Many enterprise databases have a requirement where during the regular progress of business, database rows are never deleted or updated - instead, for every change, a new row with a timestamp is added with a copy of the old data, and then mutated. In other words, many, many business applications built on top of databases have half-assed version control of the data built into them. Datomic takes the knowledge that many databases are built like this, and basically designs with that in mind. The ability to say "give me a view of the data as it was 1 month ago" is built into the system from the very beginning, rather than added on later.
- ajuc 13y ago
  There's also design pattern in systems based on relational databases that skips the need for copying so much data to keep history - instead of keeping primary data in your database with additional column "timestamp", you keep changes to that data in another table, and either use triggers triggered by inserting these changes to modify the primary data, or forgo primary data alltogether, and just use view over changes table (can be slow). I've seen it implemented 2 times in enterprise systems and it works OK, but like all design patterns it's workaround for lacking feature. Also some dbs (for example Oracle) allows you to do SELECT * FROM SOME_TABLE WHERE USER_ID = 5 AS OF TIMESTAMP..., but that's not good enough.
  - malkia 13y ago
    While looking on the subject, I do recall something about old PostgreSQL versions having something like it... But I've always wondered how they deal with changes to database schemas, types, etc. While googling I've found from this post: http://www.postgresql.org/message-id/Pine.GSO.4.64.0802031528120.7860@westnet.com http://www.postgresql.org/message-id/Pine.GSO.4.64.080203152... this free (?) book - Snodgrass/Jensen "Developing Time-Oriented Database Applications in SQL" - http://www.cs.arizona.edu/people/rts/tdbbook.pdf http://www.cs.arizona.edu/people/rts/tdbbook.pdf Haven't read it, so I'll spend some time through it. To me this is an interresting thing, since I would like to know whether I can do history tracking of game asssets for game development. E.g. often in our studio someone would like to compare data, how it was before this and what changed. We do have the mechanism, but they are not so ellegant.
    - ajuc 13y ago
      Oracle only keeps history for specified time, it takes up space, on our systems it's usualy a few hours, so it won't work for long-period history. Regarding game assets - why not keep them in git or some other cvs? For textual data it has huge additional advantage - you can blame, merge and easily see differences, but even for binary data it can be used.
gcv 13y ago
To expand on Yuri's comment: the standard view of databases, with mutable data, is flawed in the same way that mutable data structures are flawed in programming. Mutable state makes reasoning about program behavior more difficult than necessary. Datomic applies the same concept to databases — modifying a record no longer clobbers the previously-recorded information. This is a great safety feature, aside from its applicability to any auditing or reporting requirements.
- adambard 13y ago
  Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc -- I don't think you can reasonably call them "flawed." A (properly-implemented) mutable memory structure will have better performance characteristics by definition. Don't get me wrong, I like immutable data and I use it over mutable whenever possible (Clojure is my favorite language), but to call it "flawed" is an overreach at best.
  - YuriNiyazov 13y ago
    Mutable data structures are a very accurate abstraction of how computers work. They are a very inaccurate abstraction of how the world works. At any given instant, the world is constant; time moves forward, and changing the world as it was 1 second ago is not possible. Insofar as most of business programming is trying to efficiently give answers about the state of the world at various points in time, I would agree with gcv that "flawed" is the appropriate term to use for mutable data structures.
    - nnq 13y ago
      > They are a very inaccurate abstraction of how the world works. No, they are only inaccurate software abstractions of the business abstractions they are supposed to model. People like immutability and reversibility of concepts and that's why they try to apply it in business, law, finance etc. But the real/natural world, at least from our subjective perspective is not like that. Things in the world are not practically reversible or immutable - you cannot "restore" a deadbody. The real world is loosy and mutable, at least from an agent's perspective. The human brain is so efficient at what it does simply because it just "throws way* (irreversibly!) so much information every second, filtering for just what it needs. A small corner of the world is our "business/finance world" and using immutability and reversibility is great here, but there's way more to computing than implementing business/finances logic and processes in software!. ...if you think about it, classical OOP is inspired from cellular biology (can't find the reference now, some musing of Alan Kay I think...), so it's no wonder why it's such a trouble maker when used to model business logic, it was the wrong tool for the job from the get go. But generalizing that mutable data structures are a bad tool to model and understand the world in general is waaaay overstretched, the world is bigger than "business", and all modeling approaches are equally useful.
lucian1900 13y ago
Besides what has already been said, I am interested because it is one of few distributed, consistent databases capable of transactions. Google Spanner, Hyperdex and Postgres-XC are the others I know of; they appear to be quite a rare breed. There are other distributed consistent databases (HBase, RethinkDB, arguably Cassandra if you always use QUORUM, etc.) but neither supports transactions or has a design that seems to me like it easily could.
- drostie 13y ago
  Is it distributed nowadays? I vaguely remember Datomic as having a single-point-of-failure "transactor" which every request had to go through. (Given the specialized name, maybe the transactor itself doesn't have to do very much; but it still makes me wonder how you'd distribute the data store over many nodes if the nodes aren't capable of transactions themselves.)
  - lukev 13y ago
    Writes have to go through the transactor. Reads don't. So Datomic is a bit of a hybrid; writes are single point, reads are distributed. The storage layer is distributed automatically because it's decoupled from both the transactor and the peer. It's actually pluggable, too... you can use DynamoDB, Riak, Couchbase, Infinispan, Postgres, etc. which all have different performance and availability characteristics.
  - lucian1900 13y ago
    I'm ok with serialised writes for most purposes.

Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design? Honest question, I'm curious.

by Toshio 13y ago

Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design? Honest question, I'm curious.

zimbatm 13y ago

I see the created_at and updated_at fields as the poor man's datomic.

YuriNiyazov 13y ago

Many enterprise databases have a requirement where during the regular progress of business, database rows are never deleted or updated - instead, for every change, a new row with a timestamp is added with a copy of the old data, and then mutated. In other words, many, many business applications built on top of databases have half-assed version control of the data built into them. Datomic takes the knowledge that many databases are built like this, and basically designs with that in mind. The ability to say "give me a view of the data as it was 1 month ago" is built into the system from the very beginning, rather than added on later.

ajuc 13y ago
There's also design pattern in systems based on relational databases that skips the need for copying so much data to keep history - instead of keeping primary data in your database with additional column "timestamp", you keep changes to that data in another table, and either use triggers triggered by inserting these changes to modify the primary data, or forgo primary data alltogether, and just use view over changes table (can be slow). I've seen it implemented 2 times in enterprise systems and it works OK, but like all design patterns it's workaround for lacking feature. Also some dbs (for example Oracle) allows you to do SELECT * FROM SOME_TABLE WHERE USER_ID = 5 AS OF TIMESTAMP..., but that's not good enough.
- malkia 13y ago
  While looking on the subject, I do recall something about old PostgreSQL versions having something like it... But I've always wondered how they deal with changes to database schemas, types, etc. While googling I've found from this post: http://www.postgresql.org/message-id/Pine.GSO.4.64.0802031528120.7860@westnet.com http://www.postgresql.org/message-id/Pine.GSO.4.64.080203152... this free (?) book - Snodgrass/Jensen "Developing Time-Oriented Database Applications in SQL" - http://www.cs.arizona.edu/people/rts/tdbbook.pdf http://www.cs.arizona.edu/people/rts/tdbbook.pdf Haven't read it, so I'll spend some time through it. To me this is an interresting thing, since I would like to know whether I can do history tracking of game asssets for game development. E.g. often in our studio someone would like to compare data, how it was before this and what changed. We do have the mechanism, but they are not so ellegant.
  - ajuc 13y ago
    Oracle only keeps history for specified time, it takes up space, on our systems it's usualy a few hours, so it won't work for long-period history. Regarding game assets - why not keep them in git or some other cvs? For textual data it has huge additional advantage - you can blame, merge and easily see differences, but even for binary data it can be used.

gcv 13y ago

To expand on Yuri's comment: the standard view of databases, with mutable data, is flawed in the same way that mutable data structures are flawed in programming. Mutable state makes reasoning about program behavior more difficult than necessary. Datomic applies the same concept to databases — modifying a record no longer clobbers the previously-recorded information. This is a great safety feature, aside from its applicability to any auditing or reporting requirements.

adambard 13y ago
Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc -- I don't think you can reasonably call them "flawed." A (properly-implemented) mutable memory structure will have better performance characteristics by definition. Don't get me wrong, I like immutable data and I use it over mutable whenever possible (Clojure is my favorite language), but to call it "flawed" is an overreach at best.
- YuriNiyazov 13y ago
  Mutable data structures are a very accurate abstraction of how computers work. They are a very inaccurate abstraction of how the world works. At any given instant, the world is constant; time moves forward, and changing the world as it was 1 second ago is not possible. Insofar as most of business programming is trying to efficiently give answers about the state of the world at various points in time, I would agree with gcv that "flawed" is the appropriate term to use for mutable data structures.
  - nnq 13y ago
    > They are a very inaccurate abstraction of how the world works. No, they are only inaccurate software abstractions of the business abstractions they are supposed to model. People like immutability and reversibility of concepts and that's why they try to apply it in business, law, finance etc. But the real/natural world, at least from our subjective perspective is not like that. Things in the world are not practically reversible or immutable - you cannot "restore" a deadbody. The real world is loosy and mutable, at least from an agent's perspective. The human brain is so efficient at what it does simply because it just "throws way* (irreversibly!) so much information every second, filtering for just what it needs. A small corner of the world is our "business/finance world" and using immutability and reversibility is great here, but there's way more to computing than implementing business/finances logic and processes in software!. ...if you think about it, classical OOP is inspired from cellular biology (can't find the reference now, some musing of Alan Kay I think...), so it's no wonder why it's such a trouble maker when used to model business logic, it was the wrong tool for the job from the get go. But generalizing that mutable data structures are a bad tool to model and understand the world in general is waaaay overstretched, the world is bigger than "business", and all modeling approaches are equally useful.

lucian1900 13y ago

Besides what has already been said, I am interested because it is one of few distributed, consistent databases capable of transactions. Google Spanner, Hyperdex and Postgres-XC are the others I know of; they appear to be quite a rare breed. There are other distributed consistent databases (HBase, RethinkDB, arguably Cassandra if you always use QUORUM, etc.) but neither supports transactions or has a design that seems to me like it easily could.

drostie 13y ago
Is it distributed nowadays? I vaguely remember Datomic as having a single-point-of-failure "transactor" which every request had to go through. (Given the specialized name, maybe the transactor itself doesn't have to do very much; but it still makes me wonder how you'd distribute the data store over many nodes if the nodes aren't capable of transactions themselves.)
- lukev 13y ago
  Writes have to go through the transactor. Reads don't. So Datomic is a bit of a hybrid; writes are single point, reads are distributed. The storage layer is distributed automatically because it's decoupled from both the transactor and the peer. It's actually pluggable, too... you can use DynamoDB, Riak, Couchbase, Infinispan, Postgres, etc. which all have different performance and availability characteristics.
- lucian1900 13y ago
  I'm ok with serialised writes for most purposes.