9 ms·

Makes sense to see heavy usage in financial industry. Cognitect's Datomic database is perfect for the environment-datalog queries which work really well with ti

Makes sense to see heavy usage in financial industry. Cognitect's Datomic database is perfect for the environment-datalog queries which work really well with time series data. Having an immutable audit trail of all atomic operation done on the database is taking the best out of what blockchain has been marketing. It also seems like there's some demand to fund an open source version of Datomic: https://news.ycombinator.com/item?id=13583620 https://news.ycombinator.com/item?id=13583620 Crowdfund open source Datomic alternative here: http://letsopensource.com http://letsopensource.com

espeed 10y ago
NB: Datomic is not "append only" (when it was first released, I thought it was too): Note that accumulate-only is a semantic property, and is not the same as append-only, which is a structural property describing how data is written. Datomic is not an append-only system, and does not have the performance characteristics associated with append-only systems. Source: http://docs.datomic.com/indexes.html http://docs.datomic.com/indexes.html
- brilliantcode 10y ago
  I updated my original post to reflect that but I'm still not sure what the difference is.
ledgerdev 10y ago
A full open version would be a major undertaking. What I've been thinking about about is step in the datomic direction, built on top of plain old postgres that wouldn't attempt full time travel, but would use the same general schema/model with history. It's seems crazy that in 2017 that we don't have a decent general option for most business apps to store facts that doesn't over-write the past as is done in current sql databases.
- yogthos 10y ago
  This project might be of interest actually https://github.com/datacrypt-project/hitchhiker-tree https://github.com/datacrypt-project/hitchhiker-tree
  - brilliantcode 10y ago
    This one that was posted here recently seems to be pointing in the right direction. https://github.com/mozilla/mentat https://github.com/mozilla/mentat
- brilliantcode 10y ago
  Absolutely. It's a huge undertaking which is why I think it's going to take a lot of crowdfunders to get one out the door. It's not clear just how much of a demand there is for an open source alternative to Datomic which I'm trying to measure with the sign up landing page. PostgreSQL solution might work if there was a way to guarantee immutable audit trail. This is really why Datomic favored in financial industries as it gives a high level of confidence that the audit trail can't be manipulated (even if you delete a record, that action will show up in the audit trail). I do agree that it's rather surprising that we still don't have something like Datomic on github with a liberal licensing like MIT or BSD. It could be due to demand or high cost barrier associated with emulating something like Datomic. And this is what I hope crowdfunding will change the game. Pay people to work on their pet project full time by crowdfunding it.
- tom_b 10y ago
  I have had some success stealing the immutability ideas out of Datomic and sticking them into normal RDBMs. I generally add a timestamp and a deleted flag to normal entities in the schema. Updates or deletes result in new tuples with later timestamps - in the case of deletes a "deleted" column gets updated to 'Y'. All read queries are actually executed against views defined on top of the entities where the max(timestamp) is used to select the most recent instance of an entity along with a check for deleted='N'. If the most recent tuple for an instance has been deleted, it won't show up in the results. This actually works pretty well, but I have a REST API fronting the whole setup - programs are not directly accessing the underlying table or views directly.
  - shady_lady_768 10y ago
    > but I have a REST API fronting the whole setup Have overheard of this in passing. Might elaborating on setup/use-case? Seems strange to me for someone to restrict themselves artifically and not expose the whole power of SQL etc..
    - tom_b 10y ago
      Requirements were to allow internal and external teams the ability to CRUD a small set of resources using multiple programming languages, OS, and include command line scripting ability. Direct access to our HIPAA/IRB database instances are based on white-lists - most internal and all external teams are prohibited from direct database connections. So for those groups, there is no possibility of any SQL access to schemas. We use API keys with a shared secret (like AWS) to sign REST requests. If internal and external groups need to work together to jointly CRUD some resources for a specific project, they can share an API key dedicated to that project. There was also a push to switch from an app-centric view of data management to an API-driven approach serving hypermedia (collection+json). That worked conceptually, but most API users ignore the embedded hypermedia links. I find that too bad, as those links make generic API consumption clients much more resilient to change - following a link embedded in a response rather than relying on POST to a well-known URI is a useful abstraction.

espeed 10y ago

NB: Datomic is not "append only" (when it was first released, I thought it was too): Note that accumulate-only is a semantic property, and is not the same as append-only, which is a structural property describing how data is written. Datomic is not an append-only system, and does not have the performance characteristics associated with append-only systems. Source: http://docs.datomic.com/indexes.html http://docs.datomic.com/indexes.html

brilliantcode 10y ago
I updated my original post to reflect that but I'm still not sure what the difference is.

ledgerdev 10y ago

A full open version would be a major undertaking. What I've been thinking about about is step in the datomic direction, built on top of plain old postgres that wouldn't attempt full time travel, but would use the same general schema/model with history. It's seems crazy that in 2017 that we don't have a decent general option for most business apps to store facts that doesn't over-write the past as is done in current sql databases.

yogthos 10y ago
This project might be of interest actually https://github.com/datacrypt-project/hitchhiker-tree https://github.com/datacrypt-project/hitchhiker-tree
- brilliantcode 10y ago
  This one that was posted here recently seems to be pointing in the right direction. https://github.com/mozilla/mentat https://github.com/mozilla/mentat
brilliantcode 10y ago
Absolutely. It's a huge undertaking which is why I think it's going to take a lot of crowdfunders to get one out the door. It's not clear just how much of a demand there is for an open source alternative to Datomic which I'm trying to measure with the sign up landing page. PostgreSQL solution might work if there was a way to guarantee immutable audit trail. This is really why Datomic favored in financial industries as it gives a high level of confidence that the audit trail can't be manipulated (even if you delete a record, that action will show up in the audit trail). I do agree that it's rather surprising that we still don't have something like Datomic on github with a liberal licensing like MIT or BSD. It could be due to demand or high cost barrier associated with emulating something like Datomic. And this is what I hope crowdfunding will change the game. Pay people to work on their pet project full time by crowdfunding it.
tom_b 10y ago
I have had some success stealing the immutability ideas out of Datomic and sticking them into normal RDBMs. I generally add a timestamp and a deleted flag to normal entities in the schema. Updates or deletes result in new tuples with later timestamps - in the case of deletes a "deleted" column gets updated to 'Y'. All read queries are actually executed against views defined on top of the entities where the max(timestamp) is used to select the most recent instance of an entity along with a check for deleted='N'. If the most recent tuple for an instance has been deleted, it won't show up in the results. This actually works pretty well, but I have a REST API fronting the whole setup - programs are not directly accessing the underlying table or views directly.
- shady_lady_768 10y ago
  > but I have a REST API fronting the whole setup Have overheard of this in passing. Might elaborating on setup/use-case? Seems strange to me for someone to restrict themselves artifically and not expose the whole power of SQL etc..
  - tom_b 10y ago
    Requirements were to allow internal and external teams the ability to CRUD a small set of resources using multiple programming languages, OS, and include command line scripting ability. Direct access to our HIPAA/IRB database instances are based on white-lists - most internal and all external teams are prohibited from direct database connections. So for those groups, there is no possibility of any SQL access to schemas. We use API keys with a shared secret (like AWS) to sign REST requests. If internal and external groups need to work together to jointly CRUD some resources for a specific project, they can share an API key dedicated to that project. There was also a push to switch from an app-centric view of data management to an API-driven approach serving hypermedia (collection+json). That worked conceptually, but most API users ignore the embedded hypermedia links. I find that too bad, as those links make generic API consumption clients much more resilient to change - following a link embedded in a response rather than relying on POST to a well-known URI is a useful abstraction.