9 ms·

Migration from Postgres to Datomic

170 points by grzm 9y ago

modarts 9y ago
This comes up every time Datomic is mentioned, but it's tremendously sad to see such a revolutionary technology lack any sizable adoption because of its licensing model. Datomic packages up a CQRS + event-sourced architecture in a really nice way. You can stash graph/tree/relational/tabular/(insert shape here) data and query it out with the powerful Datalog language. You get time travel (versioning) for free. Etc, etc. It's a crime that this hasn't taken off as one of the leading data stores; hopefully Cognitec has some sort of roadmap to an open source release.
- itchynosedev 9y ago
  Hell, even implementing CRUD applications with Datomic / Clojure and Clojurescript is blowing everything else out of the water. No string literals, pure data. Add transit to the mix and you get full-stack killer combo with unparalleled productivity. With so many parts from Cognitect filling the stack it would be silly not to boost adoption with a friendly licensing.
  - slagfart 9y ago
    "Unparalleled productivity" sounds like hyperbole, especially considering the posts below, showing how exactly the same thing can be done trivially in Postgres. Do you have an example?
    - joe-user 9y ago
      This doesn't answer your question, but I just wanted to make clear that the posts below certainly do not show, trivially or otherwise, how to use "no string literals" and "pure data" in Postgres.
      - iaabtpbtpnn 9y ago
        Those concepts don't make any sense, so it's no wonder that a serious RDMBS like Postgres doesn't support them.
        Risord 9y ago
        Can you explain your statement?
        narag 9y ago
        Can you explain what those concepts mean and why are they important? It's the first time I've read about them or Datomic for that matter.
        Risord 9y ago
        I asked since I don't know much about datomic either or how usable is it in production. So I haven't much to share.
    - sheepmullet 9y ago
      > especially considering the posts below, showing how exactly the same thing can be done trivially in Postgres. So then where is the open source version / extension to Postgres? How about this - I'll give you $1000 if you can give me a seamless extension on top of Postgres that has most of the functionality of Datomic. Since it is trivial it is easy money for you.
      - skrebbel 9y ago
        GGP talked about building a CRUD app with "unparalleled productivity". That's where the burden of proof is, not to the GP who is skeptical about that claim.
      - hellofunk 9y ago
        You are asking for evidence of disproof. That is a logical paradox, and frequently used as an argument by extremist politicians. Where is the evidence that you do not [insert anything here]? That's a slippery slope of challenge.
        deleted 9y ago
        [deleted]
      - VLM 9y ago
        "So then where is the open source version / extension to Postgres?" As a concrete example I like the way in the datomic schema you can eliminate bridge tables for many-many relationships using :db/cardinality :db.cardinality/many and then you store or get a vector. There should be a way to have a super-SQL middleware compatible with any language bindings including a bare CLI that eats "SQL plus some cardinality" and behind the scenes implements the bridge table manipulations. Don't repeat yourself and all that, and many/many tables are kinda repetitive. There doesn't seem to be anything remotely like that out there.
    - dustingetz 9y ago
      It's because Datomic doesn't have the object/relational impedance mismatch. Because immutbility yields idealized caching so a lot of hard performance problems inherent to RDBMS go away. It's like moving from CVS to Git. Fuck it, hyperfiddle isn't launched but lets just post it here. Hyperfiddle is JSFiddle + Datomic. http://hyperfiddle.net/ http://hyperfiddle.net/ If you care about this, you should reach out, please email me!
      - vosper 9y ago
        It sounds like the site's not finished yet, but FYI it's a trainwreck on iPhone6 + Safari
      - pbowyer 9y ago
        I like the idea. Nice work!
    - moocowtruck 9y ago
      yep https://stackoverflow.com/questions/42457136/recursive-datalog-queries-for-datomic-really-slow https://stackoverflow.com/questions/42457136/recursive-datal... https://groups.google.com/forum/#!topic/datomic/insqPDtbvjk https://groups.google.com/forum/#!topic/datomic/insqPDtbvjk
  - throw13209 9y ago
    Unparalleled productivity maybe, but only once you've learned how to read code that looks like an AST, and to juggle parens in Emacs (learn paredit etc). For better or worse, history has proven that doesn't appeal to many developers. Lisps just aren't popular, and that lack of mindshare matters.
    - swlkr 9y ago
      There is parinfer which turns lisp into a white space significant language, no special commands necessary
- zcam 9y ago
  Stuart Halloway said "Datomic is not and will not be open source" in the latest defn podcast, so don't hold your breath. Hopefully they change their mind. They seem to be focused on a cloud (managed?) solution offering at the moment.
  - tosh 9y ago
    do you remember when in the episode they talk about this? edit: found it https://soundcloud.com/defn-771544745/23-the-right-honourable-stuart-halloway https://soundcloud.com/defn-771544745/23-the-right-honourabl...
  - dustingetz 9y ago
    Where can I find more about this cloud offering ????
  - _xczx 9y ago
    Thats disappointing, because Datomic is not and will not be adopted. No one wants to revolutionize data back to postgres when cognitect goes out of business. Stable solutions are found in cloud hosting from too big to fail companies and self-management. However, maybe they have found a niche through people taking on risk for major technical debt.
    - zcam 9y ago
      It probably pays the bills, I guess they're happy with it if that's still a thing after 5 years (?). It's not helping clojure grow as much as it could though and it's a bit a chicken/egg problem, more clojure users could be more datomic users and vis-versa.
    - jb1991 9y ago
      Fortunately, at least if Cognitect does go out of business, the Clojure language would survive fine. Hence why Clojure with Postgres (a very common pairing) is by far the safest bet.
    - nine_k 9y ago
      Exactly. I would only make a closed-source solution a centerpiece of my business if: * The thing is known to work for a number of other people, and uniquely solves my urgent problem, while no open solution can't do anything comparable. This is probably the Datomic niche. * The thing was around for ages, and is a cash cow of a major and reliable software maker, and is also significantly better than open software in some important area. This is how MS or IBM sell their databases; Datomic doesn't have nearly enough mindshare to compete here. * I'm hastily building an MVP that I plan to scrap anyway when the growth hits / the startup is acquired. Datomic is likely too expensive for that. By necessity, the first case holds for a small number of businesses.
- juskrey 9y ago
  $5k per year is almost free comparing to average single coder compensation.
  - vosper 9y ago
    Until they go out of business and you're fucked. That's when you wish you'd stuck with Postgres.
    - pjmlp 9y ago
      Being closed for free doesn't mean it isn't open source for those that pay. This was a common model before FOSS took off and it is still quite common in enterprise products. If company X goes out of business you get the latest source version you paid for. Having said this, I don't know if Cognitec has such kind of contracts.
      - skrebbel 9y ago
        Going off topic here, but we have an enterprise lead for our SaaS product who is asking the same. They want some sort of warranty that they can keep hosting the product themselves, should we go bust. Going open source (or open core) is currently not an option for us. Does anyone have any advice how to approach this? Example contracts, good experiences with escrow services maybe? Or bad experiences of course.
        true_religion 9y ago
        It's difficult to do for SaaS since you're not already packaging up the product. Datomic has perpetual free licenses so if they go bankrupt, you just keep the software you already have. If you had an on-prem setup or a hosted white label solution, then it'd be easier to license.
        twic 9y ago
        I worked for a company which sold hosted services in the financial sector. Escrow was a standard part of contracts. However, it wasn't taken immensely seriously; we faithfully put all of our source code into escrow once a year (or perhaps more often, but not as often as we released), but not all of the build, deployment, and infrastructure tools and config you would actually need to turn that code into a replica of the application. We didn't make any effort to "restore from backup" to verify that what was in escrow was actually any use. Customers, and their auditors, seemed happy with that. I do feel that in a really well-run operation, your build, deploy, and operation should be so automated that that's just another set of repositories to put in escrow. One way to look at it is that escrow is just another kind of disaster recovery; if you have safe copies of everything you would need to restore your operations if you lost all your current servers, then it's easy to put those in escrow too.
- mycat 9y ago
  Reminds me of Rebol. Technically as good as Lisp but arguably better as it is batteries included e.g. GUI. By the time the author made it open source it was far too late
  - vosper 9y ago
    You might be interested in the Red language, which is an open source reboot of Rebol. It's actively developed, and has been for a while, although I think there are only a handful of people (one?) working on it. http://www.red-lang.org http://www.red-lang.org
- akmiller 9y ago
  I don't understand this line of thinking. The product is outstanding and likely will be, if it is not already, the lifeblood of Cognitect as a company (in conjunction with their consulting services I'm sure). What is wrong with a company building a worthwhile product to sell? This is a community (Hacker News that is) that should be in full support of companies building great products to sell!
  - erichmond 9y ago
    I think it's fueled by a naive idea that paying for closed source software is inherently bad. Actually, it's not necessarily a naive idea, as this specific OP may have weighed the pros and cons and have decided that it's not worth the risk, but I do think there are very wrongheaded cargo cult ideas around this. We tend to gloss over the fact that when dealing with OSS: - Project maintainers can leave/change/stop supporting the software and the idea that "since it's OSS, I can just pick up the slack" sounds better in ones head, then it does in practice. - Even large projects (think Angular) can move in odd and unexpected directions, which in reality simulate a company "going out of business" for all intents and purposes. - OSS 'support' is great, until it's not. Meaning, if there are plenty of SO questions and answers, it's great, until your specific issue isn't addressed, at which point you're either looking to a third party to pay for support, or you're rolling your sleeves up and digging into a codebase you have no experience with outside of an external API level understanding. The dirty secret of large successful companies is that they actually pay for software, and the support of that software because they understand that time = money and vice versa. After using datomic in production, I'd welcome my competitor to use Postgres. I'd even push them to. It's a monstrous competitive advantage for me.
    - zcam 9y ago
      There's more than just support to paid db services, look at neo4j, datastax, elastic, mongo and cockroachdb among others.
  - nine_k 9y ago
    There is a "narrow niche" strategy, when you have a cool thing and only sell it to a small group of well-paying customers. There is an "open" strategy, when you offer the core part of your cool thing widely (cheaply / freemium model / for free), and profit from only e.g. 10% of the users that buy premium plans. You also have a fair chance to make your better way of doing things predominant, and put in the hands of a lot of people that won't pay you either way. The point is that when your cool thing may have a really wide adoption, with the "open" model you may have e.g. 50x the market share of the "narrow niche" model, and thus 5x the profit. Increased mindshare also helps further adoption. This, of course, depends on the adoption rate and paying customer rate.
- nine_k 9y ago
  The licensing policy may be unfortunate. The more important question is how many key patents are involved. That is, how feasible a comparable open-source project might be.
Cieplak 9y ago
You can achieve similar goals to Datomic using Postgres: create table snapshot ( id uuid primary key default uuid_generate_v4(), ts timestamp without time zone default now(), entity_id uuid references entity(id), value jsonb ); create table entity ( id uuid primary key default uuid_generate_v4(), snapshot_id uuid references snapshot(id) ); Rather than mutating the value of an entity, you simply append a new record to the snapshot table and update the entity's reference to it. This gives you a full audit trail of the entity.
- saurik 9y ago
  It is also an extremely trivial modification to PostgreSQL to just add that functionality: essentially you just need to disable the vacuum, reify the current transaction identifier, and allow queries to bypass MVCC (which is already storing exactly the kind of information people love to play up about Datomic: it is an append-only system that only is able to delete things due to garbage collection and the vacuum). I did a proof-of-concept of this years ago in this linked comment (which is the bottom of a long thread which involved Rich Hickey; I remember it being an interesting thread). https://news.ycombinator.com/item?id=4448767 https://news.ycombinator.com/item?id=4448767 At the time I did not provide a patch (which is weird for me) to show how "this is totally just leveraging stuff already there", but I just went and found that folder, so here is the patch: http://test.saurik.com/hackernews/snapshot_txid.diff http://test.saurik.com/hackernews/snapshot_txid.diff
  - willvarfar 9y ago
    Please please please try and get this into Postgres!
  - jtmarmon 9y ago
    Although this is definitely a part of the solution, I still see the coupling Rich mentions in his comment in the thread you linked to as an issue. To expand: when you have to query a remote database for data, you likely only want to perform one query for performance reasons (and certainly not an unbounded amount of queries - maybe two or three is acceptable but not N+1). This means that information must be passed down the call stack for what information is needed in that single or few queries, creating coupling between unrelated layers of your application. To make this more concrete: imagine you write a function to find users named Jim. At first this function is for reporting, so you just return a list of user IDs. Later, you decide to build a dashboard. You want to render all users named Jim here, but you need each of their names for display purposes. Given a remote database, you now need to modify the query function to be able to return the specific attributes you need for this use case. You can imagine if you extend the call stack this passing gets more complicated, requires merging of the queried items, etc. With datomic, since your data is in memory, your original query can stay the same, since N+1 queries are irrelevant when your data is available in RAM. I think this point is important.
brian_herman 9y ago
How does datomic compare to something like https://github.com/ApplauseOSS/djangoevents https://github.com/ApplauseOSS/djangoevents django-eventsourced?
- bjconlan 9y ago
  Only on a pseudo transactional level. (I say pseudo because I can't be 100% of how data is represented from the README.md of the djangoevents project it might infact be 1:1 to Datomic's EAVT(B) like index. They do share some commonalities such as: - history/transaction subscriptions and queries - Providing a Django ORM/Model like data access system via Datomic's 'Entity API' (although this is Django) But Datomic isn't just 'event sourcing' (although it's a way to think about solving a problem). It provides Prolog style queries, functions on the database, a GraphQL like API, Caching (Memcache) and robust Peer library (which also makes it hard for NON JVM languages to use Datomic as it uses a closed protocol... (although very likely uses Artimes/Hornet MQ's internal protocol)) I am talking from a Clojure/Datomic background (so my Django experience is old & limited) But I'm sure you could cobble together something to satisfy all these things for your stack, Datomic just does a beautiful job for Clojure (And I find Cognitect's strangle hold on the technology not great but I think they've loosened the licencing enough to make it acceptable for small businesses & startups which strikes a good balance).
hcarvalhoalves 9y ago
This is a good tutorial, but there are a few misconceptions / mistakes: > Avoid nils > > Datomic does not support nil values for attributes. When you do not have a value for an attribute, you should either skip it or pass an empty value: a zero, an empty string, etc. That’s why the most of expressions have (or "") at the end of threading macro. Datomic doesn't support `nil` on purpose: since it's a fact database, the correct approach is to not assert the attribute (iow, just omit it). Since the schema is a property of the database (not the entity), this makes your database forward-compatible to any new fact that you may need to store without having to be explicit about the past. You can also use `(missing? ?entity :attribute)` on queries, which should do a faster lookup on the EAVT index vs. checking for sentinel values. > JSON data > > In Datomic, there is no JSON type for attributes. I’m not sure I made a proper decision, but I just put those JSON data into a text attribute. Sure, where is no a way to access separate fields in a datalog query or apply roles to them. But at least I can restore the data then selecting a single entity: You can flatten a JSON object into a namespaced map. At this point you get what is essentially an entity w/ well named attributes that you can transact and query against. Since the schema is flexible and doesn't require migrations, in theory you can support even arbitrary objects that you don't know the schema in advance by inspecting the deserialized object. --- Datomic is a beast different enough to require some learning on what is optimal/idiomatic in terms of data modelling, because the paradigm shifts from appending/mutating tuples over multiple tables to asserting/retracting facts on what is essentially "one big table" (entity, attribute, value, transaction).
bm1362 9y ago
In my experience, we often ran into strange issues with Datomic- like restoring new databases with same name but different logical id, excision.. is hard, ingesting a large amount of read-only data without impacting txs, inability to have non-JVM clients- as well as the cost, opaqueness and inability to hire. We're actually going the opposite direction and wrote a replication stream to Postgres using the tx log API.
- moomin 9y ago
  This doesn't surprise me. It just plain doesn't have the eyeballs nor the deployment scale to shake all of these issues out. Postgres has both.
- dm3 9y ago
  It's definitely not without issues. However, I've found it very simple to work with once you are aware of all the operational gotchas. Needless to say, it's especially powerful in a 100% Clojure stack. The features I like most are 1) the transaction log - I've done something like that many times using Postgres and EventStore, but nothing beats the simplicity of just defining a few queries in code and having immediate updates on new transactions delivered to every peer. 2) idempotency - reasserting the same facts is a noop. Doing the same with a temporal table as suggested somewhere in this thread is not as trivial. 3) consistent db snapshots - once you get a hold of the database value, all the reads will only see the data as of time the value was retrieved. This makes application code much easier to reason about as the database can be treated as yet another immutable argument. 4) assembling a transaction value out of multiple pieces - same as the above, pure functions can all contribute to the final transaction value without having to mutate anything. 5) "free" caching on the peer - once you query something, it stays in the peer's memory. Subject to memory constraints, of course. You can definitely build something as nice as Datomic on top of Postgres, but it will take weeks to get all the details right.
  - bm1362 9y ago
    This doesn't really address anything I mentioned. The database is fine; and it was nice to work with when it was just 2 engineers. For us, it didn't scale with # of engineers, business needs like BI, ETLs of large amount of healthcare data into Datomic; then needing to delete said data. We spent a lot of time reinventing the wheel around schema management and building a declarative query interface as well. > few queries > take weeks Godspeed.
ledgerdev 9y ago
Does Event Sourcing make Datomic a moot point? Sure it can be your event log, but it seems like overkill.
- blain_the_train 9y ago
  I imagine datomic query language is significant more powerful.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
TeeWEE 9y ago
Very nice article, but in practice it almost is never usefull to switch a production system from postgres to dynamodb. Or rewrite it from python to clojure. But since its a pet project, its a nice way to learn new technology.
- JB024066 9y ago
  Interestingly, once you are on datomic you have a little more freedom of choice on underlying storage. Since Datomic uses postgres/ddb/cassandra as just a KV store it becomes trivial to switch underlying storage using the backup and restore feature. Nice if you decide to shop around or like a particular feature of a storage offering.
rcdwealth 9y ago
So you propagate me to use online services to keep my data, and trust them, instead of using PostgreSQL with more secure environment? I bet you are connected to Datomic company.
ComNik 9y ago
For anyone interested in exploring the datomic model, there is a great ClojureScript in-memory implementation called datascript (https://github.com/tonsky/datascript https://github.com/tonsky/datascript) by Nikita Prokopov.

170 points by grzm 9y ago

This comes up every time Datomic is mentioned, but it's tremendously sad to see such a revolutionary technology lack any sizable adoption because of its licensing model. Datomic packages up a CQRS + event-sourced architecture in a really nice way. You can stash graph/tree/relational/tabular/(insert shape here) data and query it out with the powerful Datalog language. You get time travel (versioning) for free. Etc, etc. It's a crime that this hasn't taken off as one of the leading data stores; hopefully Cognitec has some sort of roadmap to an open source release.

itchynosedev 9y ago
Hell, even implementing CRUD applications with Datomic / Clojure and Clojurescript is blowing everything else out of the water. No string literals, pure data. Add transit to the mix and you get full-stack killer combo with unparalleled productivity. With so many parts from Cognitect filling the stack it would be silly not to boost adoption with a friendly licensing.
- slagfart 9y ago
  "Unparalleled productivity" sounds like hyperbole, especially considering the posts below, showing how exactly the same thing can be done trivially in Postgres. Do you have an example?
  - joe-user 9y ago
    This doesn't answer your question, but I just wanted to make clear that the posts below certainly do not show, trivially or otherwise, how to use "no string literals" and "pure data" in Postgres.
    - iaabtpbtpnn 9y ago
      Those concepts don't make any sense, so it's no wonder that a serious RDMBS like Postgres doesn't support them.
      - Risord 9y ago
        Can you explain your statement?
        narag 9y ago
        Can you explain what those concepts mean and why are they important? It's the first time I've read about them or Datomic for that matter.
        Risord 9y ago
        I asked since I don't know much about datomic either or how usable is it in production. So I haven't much to share.
  - sheepmullet 9y ago
    > especially considering the posts below, showing how exactly the same thing can be done trivially in Postgres. So then where is the open source version / extension to Postgres? How about this - I'll give you $1000 if you can give me a seamless extension on top of Postgres that has most of the functionality of Datomic. Since it is trivial it is easy money for you.
    - skrebbel 9y ago
      GGP talked about building a CRUD app with "unparalleled productivity". That's where the burden of proof is, not to the GP who is skeptical about that claim.
    - hellofunk 9y ago
      You are asking for evidence of disproof. That is a logical paradox, and frequently used as an argument by extremist politicians. Where is the evidence that you do not [insert anything here]? That's a slippery slope of challenge.
      - deleted 9y ago
        [deleted]
    - VLM 9y ago
      "So then where is the open source version / extension to Postgres?" As a concrete example I like the way in the datomic schema you can eliminate bridge tables for many-many relationships using :db/cardinality :db.cardinality/many and then you store or get a vector. There should be a way to have a super-SQL middleware compatible with any language bindings including a bare CLI that eats "SQL plus some cardinality" and behind the scenes implements the bridge table manipulations. Don't repeat yourself and all that, and many/many tables are kinda repetitive. There doesn't seem to be anything remotely like that out there.
  - dustingetz 9y ago
    It's because Datomic doesn't have the object/relational impedance mismatch. Because immutbility yields idealized caching so a lot of hard performance problems inherent to RDBMS go away. It's like moving from CVS to Git. Fuck it, hyperfiddle isn't launched but lets just post it here. Hyperfiddle is JSFiddle + Datomic. http://hyperfiddle.net/ http://hyperfiddle.net/ If you care about this, you should reach out, please email me!
    - vosper 9y ago
      It sounds like the site's not finished yet, but FYI it's a trainwreck on iPhone6 + Safari
    - pbowyer 9y ago
      I like the idea. Nice work!
  - moocowtruck 9y ago
    yep https://stackoverflow.com/questions/42457136/recursive-datalog-queries-for-datomic-really-slow https://stackoverflow.com/questions/42457136/recursive-datal... https://groups.google.com/forum/#!topic/datomic/insqPDtbvjk https://groups.google.com/forum/#!topic/datomic/insqPDtbvjk
- throw13209 9y ago
  Unparalleled productivity maybe, but only once you've learned how to read code that looks like an AST, and to juggle parens in Emacs (learn paredit etc). For better or worse, history has proven that doesn't appeal to many developers. Lisps just aren't popular, and that lack of mindshare matters.
  - swlkr 9y ago
    There is parinfer which turns lisp into a white space significant language, no special commands necessary
zcam 9y ago
Stuart Halloway said "Datomic is not and will not be open source" in the latest defn podcast, so don't hold your breath. Hopefully they change their mind. They seem to be focused on a cloud (managed?) solution offering at the moment.
- tosh 9y ago
  do you remember when in the episode they talk about this? edit: found it https://soundcloud.com/defn-771544745/23-the-right-honourable-stuart-halloway https://soundcloud.com/defn-771544745/23-the-right-honourabl...
- dustingetz 9y ago
  Where can I find more about this cloud offering ????
- _xczx 9y ago
  Thats disappointing, because Datomic is not and will not be adopted. No one wants to revolutionize data back to postgres when cognitect goes out of business. Stable solutions are found in cloud hosting from too big to fail companies and self-management. However, maybe they have found a niche through people taking on risk for major technical debt.
  - zcam 9y ago
    It probably pays the bills, I guess they're happy with it if that's still a thing after 5 years (?). It's not helping clojure grow as much as it could though and it's a bit a chicken/egg problem, more clojure users could be more datomic users and vis-versa.
  - jb1991 9y ago
    Fortunately, at least if Cognitect does go out of business, the Clojure language would survive fine. Hence why Clojure with Postgres (a very common pairing) is by far the safest bet.
  - nine_k 9y ago
    Exactly. I would only make a closed-source solution a centerpiece of my business if: * The thing is known to work for a number of other people, and uniquely solves my urgent problem, while no open solution can't do anything comparable. This is probably the Datomic niche. * The thing was around for ages, and is a cash cow of a major and reliable software maker, and is also significantly better than open software in some important area. This is how MS or IBM sell their databases; Datomic doesn't have nearly enough mindshare to compete here. * I'm hastily building an MVP that I plan to scrap anyway when the growth hits / the startup is acquired. Datomic is likely too expensive for that. By necessity, the first case holds for a small number of businesses.
juskrey 9y ago
$5k per year is almost free comparing to average single coder compensation.
- vosper 9y ago
  Until they go out of business and you're fucked. That's when you wish you'd stuck with Postgres.
  - pjmlp 9y ago
    Being closed for free doesn't mean it isn't open source for those that pay. This was a common model before FOSS took off and it is still quite common in enterprise products. If company X goes out of business you get the latest source version you paid for. Having said this, I don't know if Cognitec has such kind of contracts.
    - skrebbel 9y ago
      Going off topic here, but we have an enterprise lead for our SaaS product who is asking the same. They want some sort of warranty that they can keep hosting the product themselves, should we go bust. Going open source (or open core) is currently not an option for us. Does anyone have any advice how to approach this? Example contracts, good experiences with escrow services maybe? Or bad experiences of course.
      - true_religion 9y ago
        It's difficult to do for SaaS since you're not already packaging up the product. Datomic has perpetual free licenses so if they go bankrupt, you just keep the software you already have. If you had an on-prem setup or a hosted white label solution, then it'd be easier to license.
      - twic 9y ago
        I worked for a company which sold hosted services in the financial sector. Escrow was a standard part of contracts. However, it wasn't taken immensely seriously; we faithfully put all of our source code into escrow once a year (or perhaps more often, but not as often as we released), but not all of the build, deployment, and infrastructure tools and config you would actually need to turn that code into a replica of the application. We didn't make any effort to "restore from backup" to verify that what was in escrow was actually any use. Customers, and their auditors, seemed happy with that. I do feel that in a really well-run operation, your build, deploy, and operation should be so automated that that's just another set of repositories to put in escrow. One way to look at it is that escrow is just another kind of disaster recovery; if you have safe copies of everything you would need to restore your operations if you lost all your current servers, then it's easy to put those in escrow too.
mycat 9y ago
Reminds me of Rebol. Technically as good as Lisp but arguably better as it is batteries included e.g. GUI. By the time the author made it open source it was far too late
- vosper 9y ago
  You might be interested in the Red language, which is an open source reboot of Rebol. It's actively developed, and has been for a while, although I think there are only a handful of people (one?) working on it. http://www.red-lang.org http://www.red-lang.org
akmiller 9y ago
I don't understand this line of thinking. The product is outstanding and likely will be, if it is not already, the lifeblood of Cognitect as a company (in conjunction with their consulting services I'm sure). What is wrong with a company building a worthwhile product to sell? This is a community (Hacker News that is) that should be in full support of companies building great products to sell!
- erichmond 9y ago
  I think it's fueled by a naive idea that paying for closed source software is inherently bad. Actually, it's not necessarily a naive idea, as this specific OP may have weighed the pros and cons and have decided that it's not worth the risk, but I do think there are very wrongheaded cargo cult ideas around this. We tend to gloss over the fact that when dealing with OSS: - Project maintainers can leave/change/stop supporting the software and the idea that "since it's OSS, I can just pick up the slack" sounds better in ones head, then it does in practice. - Even large projects (think Angular) can move in odd and unexpected directions, which in reality simulate a company "going out of business" for all intents and purposes. - OSS 'support' is great, until it's not. Meaning, if there are plenty of SO questions and answers, it's great, until your specific issue isn't addressed, at which point you're either looking to a third party to pay for support, or you're rolling your sleeves up and digging into a codebase you have no experience with outside of an external API level understanding. The dirty secret of large successful companies is that they actually pay for software, and the support of that software because they understand that time = money and vice versa. After using datomic in production, I'd welcome my competitor to use Postgres. I'd even push them to. It's a monstrous competitive advantage for me.
  - zcam 9y ago
    There's more than just support to paid db services, look at neo4j, datastax, elastic, mongo and cockroachdb among others.
- nine_k 9y ago
  There is a "narrow niche" strategy, when you have a cool thing and only sell it to a small group of well-paying customers. There is an "open" strategy, when you offer the core part of your cool thing widely (cheaply / freemium model / for free), and profit from only e.g. 10% of the users that buy premium plans. You also have a fair chance to make your better way of doing things predominant, and put in the hands of a lot of people that won't pay you either way. The point is that when your cool thing may have a really wide adoption, with the "open" model you may have e.g. 50x the market share of the "narrow niche" model, and thus 5x the profit. Increased mindshare also helps further adoption. This, of course, depends on the adoption rate and paying customer rate.
nine_k 9y ago
The licensing policy may be unfortunate. The more important question is how many key patents are involved. That is, how feasible a comparable open-source project might be.

Cieplak 9y ago

You can achieve similar goals to Datomic using Postgres: create table snapshot ( id uuid primary key default uuid_generate_v4(), ts timestamp without time zone default now(), entity_id uuid references entity(id), value jsonb ); create table entity ( id uuid primary key default uuid_generate_v4(), snapshot_id uuid references snapshot(id) ); Rather than mutating the value of an entity, you simply append a new record to the snapshot table and update the entity's reference to it. This gives you a full audit trail of the entity.

saurik 9y ago
It is also an extremely trivial modification to PostgreSQL to just add that functionality: essentially you just need to disable the vacuum, reify the current transaction identifier, and allow queries to bypass MVCC (which is already storing exactly the kind of information people love to play up about Datomic: it is an append-only system that only is able to delete things due to garbage collection and the vacuum). I did a proof-of-concept of this years ago in this linked comment (which is the bottom of a long thread which involved Rich Hickey; I remember it being an interesting thread). https://news.ycombinator.com/item?id=4448767 https://news.ycombinator.com/item?id=4448767 At the time I did not provide a patch (which is weird for me) to show how "this is totally just leveraging stuff already there", but I just went and found that folder, so here is the patch: http://test.saurik.com/hackernews/snapshot_txid.diff http://test.saurik.com/hackernews/snapshot_txid.diff
- willvarfar 9y ago
  Please please please try and get this into Postgres!
- jtmarmon 9y ago
  Although this is definitely a part of the solution, I still see the coupling Rich mentions in his comment in the thread you linked to as an issue. To expand: when you have to query a remote database for data, you likely only want to perform one query for performance reasons (and certainly not an unbounded amount of queries - maybe two or three is acceptable but not N+1). This means that information must be passed down the call stack for what information is needed in that single or few queries, creating coupling between unrelated layers of your application. To make this more concrete: imagine you write a function to find users named Jim. At first this function is for reporting, so you just return a list of user IDs. Later, you decide to build a dashboard. You want to render all users named Jim here, but you need each of their names for display purposes. Given a remote database, you now need to modify the query function to be able to return the specific attributes you need for this use case. You can imagine if you extend the call stack this passing gets more complicated, requires merging of the queried items, etc. With datomic, since your data is in memory, your original query can stay the same, since N+1 queries are irrelevant when your data is available in RAM. I think this point is important.

brian_herman 9y ago

How does datomic compare to something like https://github.com/ApplauseOSS/djangoevents https://github.com/ApplauseOSS/djangoevents django-eventsourced?

bjconlan 9y ago
Only on a pseudo transactional level. (I say pseudo because I can't be 100% of how data is represented from the README.md of the djangoevents project it might infact be 1:1 to Datomic's EAVT(B) like index. They do share some commonalities such as: - history/transaction subscriptions and queries - Providing a Django ORM/Model like data access system via Datomic's 'Entity API' (although this is Django) But Datomic isn't just 'event sourcing' (although it's a way to think about solving a problem). It provides Prolog style queries, functions on the database, a GraphQL like API, Caching (Memcache) and robust Peer library (which also makes it hard for NON JVM languages to use Datomic as it uses a closed protocol... (although very likely uses Artimes/Hornet MQ's internal protocol)) I am talking from a Clojure/Datomic background (so my Django experience is old & limited) But I'm sure you could cobble together something to satisfy all these things for your stack, Datomic just does a beautiful job for Clojure (And I find Cognitect's strangle hold on the technology not great but I think they've loosened the licencing enough to make it acceptable for small businesses & startups which strikes a good balance).

hcarvalhoalves 9y ago

This is a good tutorial, but there are a few misconceptions / mistakes: > Avoid nils > > Datomic does not support nil values for attributes. When you do not have a value for an attribute, you should either skip it or pass an empty value: a zero, an empty string, etc. That’s why the most of expressions have (or "") at the end of threading macro. Datomic doesn't support `nil` on purpose: since it's a fact database, the correct approach is to not assert the attribute (iow, just omit it). Since the schema is a property of the database (not the entity), this makes your database forward-compatible to any new fact that you may need to store without having to be explicit about the past. You can also use `(missing? ?entity :attribute)` on queries, which should do a faster lookup on the EAVT index vs. checking for sentinel values. > JSON data > > In Datomic, there is no JSON type for attributes. I’m not sure I made a proper decision, but I just put those JSON data into a text attribute. Sure, where is no a way to access separate fields in a datalog query or apply roles to them. But at least I can restore the data then selecting a single entity: You can flatten a JSON object into a namespaced map. At this point you get what is essentially an entity w/ well named attributes that you can transact and query against. Since the schema is flexible and doesn't require migrations, in theory you can support even arbitrary objects that you don't know the schema in advance by inspecting the deserialized object. --- Datomic is a beast different enough to require some learning on what is optimal/idiomatic in terms of data modelling, because the paradigm shifts from appending/mutating tuples over multiple tables to asserting/retracting facts on what is essentially "one big table" (entity, attribute, value, transaction).

bm1362 9y ago

In my experience, we often ran into strange issues with Datomic- like restoring new databases with same name but different logical id, excision.. is hard, ingesting a large amount of read-only data without impacting txs, inability to have non-JVM clients- as well as the cost, opaqueness and inability to hire. We're actually going the opposite direction and wrote a replication stream to Postgres using the tx log API.

moomin 9y ago
This doesn't surprise me. It just plain doesn't have the eyeballs nor the deployment scale to shake all of these issues out. Postgres has both.
dm3 9y ago
It's definitely not without issues. However, I've found it very simple to work with once you are aware of all the operational gotchas. Needless to say, it's especially powerful in a 100% Clojure stack. The features I like most are 1) the transaction log - I've done something like that many times using Postgres and EventStore, but nothing beats the simplicity of just defining a few queries in code and having immediate updates on new transactions delivered to every peer. 2) idempotency - reasserting the same facts is a noop. Doing the same with a temporal table as suggested somewhere in this thread is not as trivial. 3) consistent db snapshots - once you get a hold of the database value, all the reads will only see the data as of time the value was retrieved. This makes application code much easier to reason about as the database can be treated as yet another immutable argument. 4) assembling a transaction value out of multiple pieces - same as the above, pure functions can all contribute to the final transaction value without having to mutate anything. 5) "free" caching on the peer - once you query something, it stays in the peer's memory. Subject to memory constraints, of course. You can definitely build something as nice as Datomic on top of Postgres, but it will take weeks to get all the details right.
- bm1362 9y ago
  This doesn't really address anything I mentioned. The database is fine; and it was nice to work with when it was just 2 engineers. For us, it didn't scale with # of engineers, business needs like BI, ETLs of large amount of healthcare data into Datomic; then needing to delete said data. We spent a lot of time reinventing the wheel around schema management and building a declarative query interface as well. > few queries > take weeks Godspeed.

ledgerdev 9y ago

Does Event Sourcing make Datomic a moot point? Sure it can be your event log, but it seems like overkill.

blain_the_train 9y ago
I imagine datomic query language is significant more powerful.

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

TeeWEE 9y ago

Very nice article, but in practice it almost is never usefull to switch a production system from postgres to dynamodb. Or rewrite it from python to clojure. But since its a pet project, its a nice way to learn new technology.

JB024066 9y ago
Interestingly, once you are on datomic you have a little more freedom of choice on underlying storage. Since Datomic uses postgres/ddb/cassandra as just a KV store it becomes trivial to switch underlying storage using the backup and restore feature. Nice if you decide to shop around or like a particular feature of a storage offering.

rcdwealth 9y ago

So you propagate me to use online services to keep my data, and trust them, instead of using PostgreSQL with more secure environment? I bet you are connected to Datomic company.

ComNik 9y ago

For anyone interested in exploring the datomic model, there is a great ClojureScript in-memory implementation called datascript (https://github.com/tonsky/datascript https://github.com/tonsky/datascript) by Nikita Prokopov.