7 ms·

Excision in Datomic

69 points by fogus 13y ago

lucian1900 13y ago
This is (was?) one of the reasons I don't use Datomic. The other is that it's closed-source, which sadly has not changed. I certainly wouldn't mind paying for commercial usage/support. I can't justify relying on something I have no control over whatsoever when there are so many open source alternatives, however clean and nice I find Datomic's design to be :(
- Toshio 13y ago
  Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design? Honest question, I'm curious.
  - zimbatm 13y ago
    I see the created_at and updated_at fields as the poor man's datomic.
  - YuriNiyazov 13y ago
    Many enterprise databases have a requirement where during the regular progress of business, database rows are never deleted or updated - instead, for every change, a new row with a timestamp is added with a copy of the old data, and then mutated. In other words, many, many business applications built on top of databases have half-assed version control of the data built into them. Datomic takes the knowledge that many databases are built like this, and basically designs with that in mind. The ability to say "give me a view of the data as it was 1 month ago" is built into the system from the very beginning, rather than added on later.
    - ajuc 13y ago
      There's also design pattern in systems based on relational databases that skips the need for copying so much data to keep history - instead of keeping primary data in your database with additional column "timestamp", you keep changes to that data in another table, and either use triggers triggered by inserting these changes to modify the primary data, or forgo primary data alltogether, and just use view over changes table (can be slow). I've seen it implemented 2 times in enterprise systems and it works OK, but like all design patterns it's workaround for lacking feature. Also some dbs (for example Oracle) allows you to do SELECT * FROM SOME_TABLE WHERE USER_ID = 5 AS OF TIMESTAMP..., but that's not good enough.
      - malkia 13y ago
        While looking on the subject, I do recall something about old PostgreSQL versions having something like it... But I've always wondered how they deal with changes to database schemas, types, etc. While googling I've found from this post: http://www.postgresql.org/message-id/Pine.GSO.4.64.0802031528120.7860@westnet.com http://www.postgresql.org/message-id/Pine.GSO.4.64.080203152... this free (?) book - Snodgrass/Jensen "Developing Time-Oriented Database Applications in SQL" - http://www.cs.arizona.edu/people/rts/tdbbook.pdf http://www.cs.arizona.edu/people/rts/tdbbook.pdf Haven't read it, so I'll spend some time through it. To me this is an interresting thing, since I would like to know whether I can do history tracking of game asssets for game development. E.g. often in our studio someone would like to compare data, how it was before this and what changed. We do have the mechanism, but they are not so ellegant.
        ajuc 13y ago
        Oracle only keeps history for specified time, it takes up space, on our systems it's usualy a few hours, so it won't work for long-period history. Regarding game assets - why not keep them in git or some other cvs? For textual data it has huge additional advantage - you can blame, merge and easily see differences, but even for binary data it can be used.
  - gcv 13y ago
    To expand on Yuri's comment: the standard view of databases, with mutable data, is flawed in the same way that mutable data structures are flawed in programming. Mutable state makes reasoning about program behavior more difficult than necessary. Datomic applies the same concept to databases — modifying a record no longer clobbers the previously-recorded information. This is a great safety feature, aside from its applicability to any auditing or reporting requirements.
    - adambard 13y ago
      Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc -- I don't think you can reasonably call them "flawed." A (properly-implemented) mutable memory structure will have better performance characteristics by definition. Don't get me wrong, I like immutable data and I use it over mutable whenever possible (Clojure is my favorite language), but to call it "flawed" is an overreach at best.
      - YuriNiyazov 13y ago
        Mutable data structures are a very accurate abstraction of how computers work. They are a very inaccurate abstraction of how the world works. At any given instant, the world is constant; time moves forward, and changing the world as it was 1 second ago is not possible. Insofar as most of business programming is trying to efficiently give answers about the state of the world at various points in time, I would agree with gcv that "flawed" is the appropriate term to use for mutable data structures.
        nnq 13y ago
        > They are a very inaccurate abstraction of how the world works. No, they are only inaccurate software abstractions of the business abstractions they are supposed to model. People like immutability and reversibility of concepts and that's why they try to apply it in business, law, finance etc. But the real/natural world, at least from our subjective perspective is not like that. Things in the world are not practically reversible or immutable - you cannot "restore" a deadbody. The real world is loosy and mutable, at least from an agent's perspective. The human brain is so efficient at what it does simply because it just "throws way* (irreversibly!) so much information every second, filtering for just what it needs. A small corner of the world is our "business/finance world" and using immutability and reversibility is great here, but there's way more to computing than implementing business/finances logic and processes in software!. ...if you think about it, classical OOP is inspired from cellular biology (can't find the reference now, some musing of Alan Kay I think...), so it's no wonder why it's such a trouble maker when used to model business logic, it was the wrong tool for the job from the get go. But generalizing that mutable data structures are a bad tool to model and understand the world in general is waaaay overstretched, the world is bigger than "business", and all modeling approaches are equally useful.
  - lucian1900 13y ago
    Besides what has already been said, I am interested because it is one of few distributed, consistent databases capable of transactions. Google Spanner, Hyperdex and Postgres-XC are the others I know of; they appear to be quite a rare breed. There are other distributed consistent databases (HBase, RethinkDB, arguably Cassandra if you always use QUORUM, etc.) but neither supports transactions or has a design that seems to me like it easily could.
    - drostie 13y ago
      Is it distributed nowadays? I vaguely remember Datomic as having a single-point-of-failure "transactor" which every request had to go through. (Given the specialized name, maybe the transactor itself doesn't have to do very much; but it still makes me wonder how you'd distribute the data store over many nodes if the nodes aren't capable of transactions themselves.)
      - lukev 13y ago
        Writes have to go through the transactor. Reads don't. So Datomic is a bit of a hybrid; writes are single point, reads are distributed. The storage layer is distributed automatically because it's decoupled from both the transactor and the peer. It's actually pluggable, too... you can use DynamoDB, Riak, Couchbase, Infinispan, Postgres, etc. which all have different performance and availability characteristics.
      - lucian1900 13y ago
        I'm ok with serialised writes for most purposes.
- CurrentB 13y ago
  Every time datomic is brought up, people always chime in with this whining about it being closed source and I just find it sounds entitled and petty. It makes it seem like with enough people agreeing with you and enough upvotes or whatever, you think the releasers of datomic might reconsider and open source their code. This is the same person/people that released clojure to the world (as open source). It's not like they're a company with a history of closed proprietary software, they are fully aware of their options, and I'm sure they have their reasons for keeping it closed. Aren't they entitled to do whatever they want with the code they produce without frowny faces all over the comment threads?
  - necubi 13y ago
    DBs are generally a hugely crucial part of your infrastructure, and it makes sense to be conservative with them. Part of that conservatism is a heavy bias toward open source. Especially given a heavily idiosyncratic database like datomic, with large switching costs to anything else, relying on a proprietary product that could disappear at any time is a huge risk factor for your business. This is why the business model of nearly every NoSQL business is to provide commercial support for their open source DB. Examples include Basho (Riak), Datastax (Cassandra), 10gen (MongoDB), RethinkDB, VoltDB and Couchbase. There are a few exceptions (mostly in the very large data space), but the general rule is: if you're promoting a new database, make it open source and sell services/enhancements around it.
    - calibraxis 13y ago
      In my experience, corporations and most programmers love closed-source software and services. Like Amazon AWS (and everything in it like S3, DynamoDB, etc), GitHub, MacOS X, whatever. But if it's free, they'll certainly freeload on that too. There are advantages to the opensource strategy, but there's tradeoffs. Programming languages are different; Clojure likely wouldn't have succeeded were it closed-source. (BTW, Basho and Datastax seem to have "enterprise" products. Are these closed-source or contain hidden closed-services? And we should keep in mind that DBs like Riak and Hadoop are clones of closed-source DBs. Basho employees even mention that the authors of the Dynamo paper were probably nearly fired for releasing it.)
darklajid 13y ago
Another 'broken on mobile' site (Aurora on Android here). The tech might be cool, the site is awful. Downvoters: I like Datomic, like Stuart, but the site is crap. First visit had lots of artifacts (content was seemingly above the annoying header), cannot reproduce that now. Right now a huge part of the display is a fixed header, the content is 'unzoomable'. There's no way to say anything nice about that sort of stuff. http://db.tt/sIstWJhZ http://db.tt/sIstWJhZ
- samolang 13y ago
  Looks great on the default android browser.
- pasbesoin 13y ago
  One of the craptastic Blogger templates. I've taken to viewing them through Google Cache, when possible (and when I'm sufficiently motivated to not simply quit the item/site); unfortunately, more and more seem to be excluding themselves from said caching. http://webcache.googleusercontent.com/search?q=cache:http%3A%2F%2Fblog.datomic.com%2F2013%2F05%2Fexcision.html http://webcache.googleusercontent.com/search?q=cache:http%3A...
  - darklajid 13y ago
    Hmm. It seems Google Cache doesn't help me here. Now I was able to reproduce the artifacts in the page header, but the huge header itself remains (too much screen space, literally zero value) and the content still fails to allow pinch/zoom to actually read the text in the illustrations. http://db.tt/Yz5J6zd4 http://db.tt/Yz5J6zd4
abraininavat 13y ago
It is a key value proposition of Datomic Is that a purposeful pun? Because it confused me.

69 points by fogus 13y ago

lucian1900 13y ago

This is (was?) one of the reasons I don't use Datomic. The other is that it's closed-source, which sadly has not changed. I certainly wouldn't mind paying for commercial usage/support. I can't justify relying on something I have no control over whatsoever when there are so many open source alternatives, however clean and nice I find Datomic's design to be :(

Toshio 13y ago
Supposing Datomic was open-source. Can you give a short summary of what you find so appealing about its design? Honest question, I'm curious.
- zimbatm 13y ago
  I see the created_at and updated_at fields as the poor man's datomic.
- YuriNiyazov 13y ago
  Many enterprise databases have a requirement where during the regular progress of business, database rows are never deleted or updated - instead, for every change, a new row with a timestamp is added with a copy of the old data, and then mutated. In other words, many, many business applications built on top of databases have half-assed version control of the data built into them. Datomic takes the knowledge that many databases are built like this, and basically designs with that in mind. The ability to say "give me a view of the data as it was 1 month ago" is built into the system from the very beginning, rather than added on later.
  - ajuc 13y ago
    There's also design pattern in systems based on relational databases that skips the need for copying so much data to keep history - instead of keeping primary data in your database with additional column "timestamp", you keep changes to that data in another table, and either use triggers triggered by inserting these changes to modify the primary data, or forgo primary data alltogether, and just use view over changes table (can be slow). I've seen it implemented 2 times in enterprise systems and it works OK, but like all design patterns it's workaround for lacking feature. Also some dbs (for example Oracle) allows you to do SELECT * FROM SOME_TABLE WHERE USER_ID = 5 AS OF TIMESTAMP..., but that's not good enough.
    - malkia 13y ago
      While looking on the subject, I do recall something about old PostgreSQL versions having something like it... But I've always wondered how they deal with changes to database schemas, types, etc. While googling I've found from this post: http://www.postgresql.org/message-id/Pine.GSO.4.64.0802031528120.7860@westnet.com http://www.postgresql.org/message-id/Pine.GSO.4.64.080203152... this free (?) book - Snodgrass/Jensen "Developing Time-Oriented Database Applications in SQL" - http://www.cs.arizona.edu/people/rts/tdbbook.pdf http://www.cs.arizona.edu/people/rts/tdbbook.pdf Haven't read it, so I'll spend some time through it. To me this is an interresting thing, since I would like to know whether I can do history tracking of game asssets for game development. E.g. often in our studio someone would like to compare data, how it was before this and what changed. We do have the mechanism, but they are not so ellegant.
      - ajuc 13y ago
        Oracle only keeps history for specified time, it takes up space, on our systems it's usualy a few hours, so it won't work for long-period history. Regarding game assets - why not keep them in git or some other cvs? For textual data it has huge additional advantage - you can blame, merge and easily see differences, but even for binary data it can be used.
- gcv 13y ago
  To expand on Yuri's comment: the standard view of databases, with mutable data, is flawed in the same way that mutable data structures are flawed in programming. Mutable state makes reasoning about program behavior more difficult than necessary. Datomic applies the same concept to databases — modifying a record no longer clobbers the previously-recorded information. This is a great safety feature, aside from its applicability to any auditing or reporting requirements.
  - adambard 13y ago
    Mutable data structures are a much more accurate abstraction when it comes to how computers actually store data in memory and on disc -- I don't think you can reasonably call them "flawed." A (properly-implemented) mutable memory structure will have better performance characteristics by definition. Don't get me wrong, I like immutable data and I use it over mutable whenever possible (Clojure is my favorite language), but to call it "flawed" is an overreach at best.
    - YuriNiyazov 13y ago
      Mutable data structures are a very accurate abstraction of how computers work. They are a very inaccurate abstraction of how the world works. At any given instant, the world is constant; time moves forward, and changing the world as it was 1 second ago is not possible. Insofar as most of business programming is trying to efficiently give answers about the state of the world at various points in time, I would agree with gcv that "flawed" is the appropriate term to use for mutable data structures.
      - nnq 13y ago
        > They are a very inaccurate abstraction of how the world works. No, they are only inaccurate software abstractions of the business abstractions they are supposed to model. People like immutability and reversibility of concepts and that's why they try to apply it in business, law, finance etc. But the real/natural world, at least from our subjective perspective is not like that. Things in the world are not practically reversible or immutable - you cannot "restore" a deadbody. The real world is loosy and mutable, at least from an agent's perspective. The human brain is so efficient at what it does simply because it just "throws way* (irreversibly!) so much information every second, filtering for just what it needs. A small corner of the world is our "business/finance world" and using immutability and reversibility is great here, but there's way more to computing than implementing business/finances logic and processes in software!. ...if you think about it, classical OOP is inspired from cellular biology (can't find the reference now, some musing of Alan Kay I think...), so it's no wonder why it's such a trouble maker when used to model business logic, it was the wrong tool for the job from the get go. But generalizing that mutable data structures are a bad tool to model and understand the world in general is waaaay overstretched, the world is bigger than "business", and all modeling approaches are equally useful.
- lucian1900 13y ago
  Besides what has already been said, I am interested because it is one of few distributed, consistent databases capable of transactions. Google Spanner, Hyperdex and Postgres-XC are the others I know of; they appear to be quite a rare breed. There are other distributed consistent databases (HBase, RethinkDB, arguably Cassandra if you always use QUORUM, etc.) but neither supports transactions or has a design that seems to me like it easily could.
  - drostie 13y ago
    Is it distributed nowadays? I vaguely remember Datomic as having a single-point-of-failure "transactor" which every request had to go through. (Given the specialized name, maybe the transactor itself doesn't have to do very much; but it still makes me wonder how you'd distribute the data store over many nodes if the nodes aren't capable of transactions themselves.)
    - lukev 13y ago
      Writes have to go through the transactor. Reads don't. So Datomic is a bit of a hybrid; writes are single point, reads are distributed. The storage layer is distributed automatically because it's decoupled from both the transactor and the peer. It's actually pluggable, too... you can use DynamoDB, Riak, Couchbase, Infinispan, Postgres, etc. which all have different performance and availability characteristics.
    - lucian1900 13y ago
      I'm ok with serialised writes for most purposes.
CurrentB 13y ago
Every time datomic is brought up, people always chime in with this whining about it being closed source and I just find it sounds entitled and petty. It makes it seem like with enough people agreeing with you and enough upvotes or whatever, you think the releasers of datomic might reconsider and open source their code. This is the same person/people that released clojure to the world (as open source). It's not like they're a company with a history of closed proprietary software, they are fully aware of their options, and I'm sure they have their reasons for keeping it closed. Aren't they entitled to do whatever they want with the code they produce without frowny faces all over the comment threads?
- necubi 13y ago
  DBs are generally a hugely crucial part of your infrastructure, and it makes sense to be conservative with them. Part of that conservatism is a heavy bias toward open source. Especially given a heavily idiosyncratic database like datomic, with large switching costs to anything else, relying on a proprietary product that could disappear at any time is a huge risk factor for your business. This is why the business model of nearly every NoSQL business is to provide commercial support for their open source DB. Examples include Basho (Riak), Datastax (Cassandra), 10gen (MongoDB), RethinkDB, VoltDB and Couchbase. There are a few exceptions (mostly in the very large data space), but the general rule is: if you're promoting a new database, make it open source and sell services/enhancements around it.
  - calibraxis 13y ago
    In my experience, corporations and most programmers love closed-source software and services. Like Amazon AWS (and everything in it like S3, DynamoDB, etc), GitHub, MacOS X, whatever. But if it's free, they'll certainly freeload on that too. There are advantages to the opensource strategy, but there's tradeoffs. Programming languages are different; Clojure likely wouldn't have succeeded were it closed-source. (BTW, Basho and Datastax seem to have "enterprise" products. Are these closed-source or contain hidden closed-services? And we should keep in mind that DBs like Riak and Hadoop are clones of closed-source DBs. Basho employees even mention that the authors of the Dynamo paper were probably nearly fired for releasing it.)

darklajid 13y ago

Another 'broken on mobile' site (Aurora on Android here). The tech might be cool, the site is awful. Downvoters: I like Datomic, like Stuart, but the site is crap. First visit had lots of artifacts (content was seemingly above the annoying header), cannot reproduce that now. Right now a huge part of the display is a fixed header, the content is 'unzoomable'. There's no way to say anything nice about that sort of stuff. http://db.tt/sIstWJhZ http://db.tt/sIstWJhZ

samolang 13y ago
Looks great on the default android browser.
pasbesoin 13y ago
One of the craptastic Blogger templates. I've taken to viewing them through Google Cache, when possible (and when I'm sufficiently motivated to not simply quit the item/site); unfortunately, more and more seem to be excluding themselves from said caching. http://webcache.googleusercontent.com/search?q=cache:http%3A%2F%2Fblog.datomic.com%2F2013%2F05%2Fexcision.html http://webcache.googleusercontent.com/search?q=cache:http%3A...
- darklajid 13y ago
  Hmm. It seems Google Cache doesn't help me here. Now I was able to reproduce the artifacts in the page header, but the huge header itself remains (too much screen space, literally zero value) and the content still fails to allow pinch/zoom to actually read the text in the illustrations. http://db.tt/Yz5J6zd4 http://db.tt/Yz5J6zd4

abraininavat 13y ago

It is a key value proposition of Datomic Is that a purposeful pun? Because it confused me.