5 ms·

From experience: Datomic Cloud is slow, expensive, resource intensive, designed in the baroque style of massively over-complicated CloudFormation astronautics.

From experience: Datomic Cloud is slow, expensive, resource intensive, designed in the baroque style of massively over-complicated CloudFormation astronautics. Hard to diagnose performance issues. Impossible to backup. Ran into one scenario where apparently we weren't quick enough to migrate to the latest version, AWS had dropped support for $runtime in Lambda, and it became impossible to upgrade the CloudFormation template. Had to write application code to export/reimport prod data from cluster to another—there was no other migration path (and yes, we were talking to their enterprise support). We migrated to Postgres and are now using a 10th of the compute resources. Our p99 response times went from 1.3-1.5s to under 300ms once all the read traffic was cut over. Mother Postgres can do no wrong. Still, Datomic seems like a cool idea.

panick21_ 3y ago
They should have just focused on having a great Kubernetes setup experience, their focus on this cloud stuff always seemed strange to me.
JulianWasTaken 3y ago
I don't doubt your story of course, and I love Postgres, but comparing apples to oranges no? Datomic's killer feature is time travel. Did you simply not use that feature once you moved off Datomic (and if so why'd you pick Datomic in the first place)? Or are you using Postgres using some extension to add in?
- bvanderveen 3y ago
  We implemented it in Postgres with 'created_at' and 'deleted_at' columns on everything and filtering to make sure that the object 'exists' at the time the query is concerned with. Changes in relationships between objects are modeled as join tables with a boolean indicating whether the relationship is made or broken and at what time. Our data model is not large and we had a very complete test suite already, so it was easy to produce another implementation backed by postgres, RAM, etc.
  - wwweston 3y ago
    Yeah, it seems you could be able to substitute thoughtful schema design avoiding updates/deletes for time-travel as a feature. I wonder if anyone has made a collection of reference examples implemented this way (and in general think that a substantial compendium good examples of DB schema and thinking behind them could be worthwhile).
    - mjul 3y ago
      Snodgrass wrote a whole book on that topic: Developing Time-Oriented Database Applications in SQL (1999). It is available as PDF on his publications page: https://www2.cs.arizona.edu/~rts/publications.html https://www2.cs.arizona.edu/~rts/publications.html
    - jskulski 3y ago
      Ive built a couple systems that would have been datomic’s bread and butter. Each time the company was more comfortable with mainstream dbs, so we ended going with something like you’re talking about, built on top of a db. A couple of the projects were because a mainstream dbs wouldn’t scale. The systems definitely worked, but it was also a lot of implementation complexity on an other wise simple business prop: “store this data as facts”
    - twic 3y ago
      I'm moderately confident you could mechanically transform a time-oblivious schema into a history-preserving one, and then write a view on top of it which gave a slice at a particular time. Moderately.
      - ysleepy 3y ago
        That is essentially what MVCC does.
        twic 3y ago
        That is an extremely good point.
- ithrow 3y ago
  Datomic's 'time travel' is an audit feature, not something for your application/business logic to depend on. Performance reasons make it impractical, unless you only have like 10 users and very little data.
  - JulianWasTaken 3y ago
    That's certainly not how it sells and markets itself. The first feature on benefits (and the only reason I've ever heard Datomic brought up and/or considered it myself for production workflows) is using that stuff in application workflows: https://docs.datomic.com/pro/time/filters.html#history https://docs.datomic.com/pro/time/filters.html#history Could be you're saying it in fact doesn't work well performance-wise, that'd (surprise me but) certainly explain why it's not more popular -- but I think it's clear it wants you to use this as an application feature.
    - newlisp 3y ago
      Welcome to sales tactics ;) Datomic is great but as another commenter said, is good for "small-ish backoffice systems that never has to be web scale". You almost probably can rely on querying history for internal applications. I think their primary market was for companies to use it internally but they never made this clear.
      - KingMob 3y ago
        Ironically, Hickey fired the one marketer they hired for Datomic. He lucked out when a unicorn went all in on it. Word around Cognitect was, Datomic was barely breaking even.
      - xmlblog 3y ago
        > "small-ish backoffice systems that never has to be web scale". Doesn't production use of Datomic by Nubank and Netflix (to mention just two examples) belie this assertion?
        bilkow 3y ago
- btown 3y ago
  https://www.postgresql.org/docs/11/contrib-spi.html#id-1.11.7.45.6 https://www.postgresql.org/docs/11/contrib-spi.html#id-1.11.... discusses a model for implementing time travel in Postgres <12 using SPI. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2d10defa7771931295ee804cf096fb9388472a7b https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... discusses why it was removed in Postgres 12 - it seems logical that it's more maintainable to implement in plpgsql, though as far as I can tell there aren't off-the-shelf implementations of this. We use https://django-simple-history.readthedocs.io/en/latest/ https://django-simple-history.readthedocs.io/en/latest/ (with some custom tooling for diff generation) for audit logs and resettability, and while you can't move an entire set of tables back in time simultaneously, it's usually sufficient for understanding data history.
ithrow 3y ago
I guess this is why datomic.com front page now defaults to datomic pro and not cloud.
avodonosov 3y ago
Why backups were impossible? Couldn't you backed up the storage resources?
- bvanderveen 3y ago
  As far as I could tell, there was no straightforward way to point a new instance of the compute resources back at the old storage resources since they're all provisioned in one CF template.
outworlder 3y ago
Are they _forcing_ you to use CloudFormation? Or is it just the officially supported mechanism? > Mother Postgres can do no wrong. I'll say that Postgres is usually the answer for the vast majority of use-cases. Even when you think you need something else to do something different, it's probably still a good enough solution. I've seen teams pitching other system just because they wanted to push a bunch of JSON. Guess what, PG can handle that fine and even run SQL queries against that. PG can access other database systems with its foreign data wrappers(https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers). The main difficulty is that horizontally scaling it is not trivial(although not impossible, and that can be improved with third party companies).
- JohnBooty 3y ago
  Yes. Postgres such a reliable and known quantity that IMO it should be the default choice for just about anything. Don't misunderstand me. There are plenty of times when something else is the right choice. I'm just saying, when I have a say in the matter, folks need to clear that bar -- "tell me why tool xyz is going to be so much better than postgres for this use case that it justifies the overhead of adding another piece of software infrastructure." Like, you want to add a document database? Obviously Mongo, Elasticsearch, etc are "best of breed." But Postgres is pretty capable and this team is already good at it. Are we ever going to have so many documents that e.g. Elasticsearch's mostly-effortless horizontal scaling even comes into play? If you don't ever see yourself scaling past 1,000 documents then adding a new piece of infra is a total joke. I see that kind of thing all the time. I can't tell if developers truly do not understand scale, or if they simply do not give a f--- and simply want to play with shiny new toys and enrich their resumes. I mean, I've literally had devops guys telling me we need a Redis cluster even though we were only storing a few kilobytes of data, that was read dozens of times daily with zero plans to scale. That could have been a f'in Postgres table. Devops guy defended that choice hard even when pressed by mgmt to reduce AWS spend. WTF?
  - jwr 3y ago
    > Postgres such a reliable and known quantity that IMO it should be the default choice for just about anything. This is being repeated so often. And yet — the above is true, IF (and that's a big if for some of us) you are OK with having your database on a single machine. If you want a distributed database with strict serializability, where some nodes can go down and you still get correct answers, Postgres is not it.
lgrapenthin 3y ago
As someone who is using Datomic Pro in production for many years now I must agree with you. One time I began a project with Datomic Cloud and it was a disaster similar to what you described. I learned a lot about AWS, but after about half a year we switched to Datomic Pro. There were some cool ideas in Datomic Cloud, like IONs and its integrated deployment CLI. But the dev workflow with Datomic Pro in the REPL, potentially connected to your live or staging database is much more interactive and fun than waiting for CodeDeploy. I guess there is a reason Datomic Pro is the featured product on datomic.com again. It appears that Cognitect took a big bet with Datomic Cloud and it didn't take off. Soon after the NuBank acquisition happened. That being said, Datomic Cloud was not a bad idea, it just turned out that Datomic Pro/onPrem is much easier to use. Also of all their APIs, the "Peer API" of Pro is just the best IME, especially with `d/entity` vs. "pull" etc.
ggleason 3y ago
> Datomic Cloud is slow, expensive, resource intensive, designed in the baroque style of massively over-complicated CloudFormation astronautics. Hard to diagnose performance issues. Impossible to backup. You should give TerminusDB a go (https://terminusdb.com/ https://terminusdb.com/), it's really OSS, the cloud version is cheap, fast, there are not tons of baroque settings, and it's easy to backup using clone. TermiusDB is a graph database with a git-like model with push/pull/clone semantics as well as a datalog.
froggertoaster 3y ago
> Mother Postgres can do no wrong. Simple, eloquent, damn true.

panick21_ 3y ago

They should have just focused on having a great Kubernetes setup experience, their focus on this cloud stuff always seemed strange to me.

JulianWasTaken 3y ago

I don't doubt your story of course, and I love Postgres, but comparing apples to oranges no? Datomic's killer feature is time travel. Did you simply not use that feature once you moved off Datomic (and if so why'd you pick Datomic in the first place)? Or are you using Postgres using some extension to add in?

bvanderveen 3y ago
We implemented it in Postgres with 'created_at' and 'deleted_at' columns on everything and filtering to make sure that the object 'exists' at the time the query is concerned with. Changes in relationships between objects are modeled as join tables with a boolean indicating whether the relationship is made or broken and at what time. Our data model is not large and we had a very complete test suite already, so it was easy to produce another implementation backed by postgres, RAM, etc.
- wwweston 3y ago
  Yeah, it seems you could be able to substitute thoughtful schema design avoiding updates/deletes for time-travel as a feature. I wonder if anyone has made a collection of reference examples implemented this way (and in general think that a substantial compendium good examples of DB schema and thinking behind them could be worthwhile).
  - mjul 3y ago
    Snodgrass wrote a whole book on that topic: Developing Time-Oriented Database Applications in SQL (1999). It is available as PDF on his publications page: https://www2.cs.arizona.edu/~rts/publications.html https://www2.cs.arizona.edu/~rts/publications.html
  - jskulski 3y ago
    Ive built a couple systems that would have been datomic’s bread and butter. Each time the company was more comfortable with mainstream dbs, so we ended going with something like you’re talking about, built on top of a db. A couple of the projects were because a mainstream dbs wouldn’t scale. The systems definitely worked, but it was also a lot of implementation complexity on an other wise simple business prop: “store this data as facts”
  - twic 3y ago
    I'm moderately confident you could mechanically transform a time-oblivious schema into a history-preserving one, and then write a view on top of it which gave a slice at a particular time. Moderately.
    - ysleepy 3y ago
      That is essentially what MVCC does.
      - twic 3y ago
        That is an extremely good point.
ithrow 3y ago
Datomic's 'time travel' is an audit feature, not something for your application/business logic to depend on. Performance reasons make it impractical, unless you only have like 10 users and very little data.
- JulianWasTaken 3y ago
  That's certainly not how it sells and markets itself. The first feature on benefits (and the only reason I've ever heard Datomic brought up and/or considered it myself for production workflows) is using that stuff in application workflows: https://docs.datomic.com/pro/time/filters.html#history https://docs.datomic.com/pro/time/filters.html#history Could be you're saying it in fact doesn't work well performance-wise, that'd (surprise me but) certainly explain why it's not more popular -- but I think it's clear it wants you to use this as an application feature.
  - newlisp 3y ago
    Welcome to sales tactics ;) Datomic is great but as another commenter said, is good for "small-ish backoffice systems that never has to be web scale". You almost probably can rely on querying history for internal applications. I think their primary market was for companies to use it internally but they never made this clear.
    - KingMob 3y ago
      Ironically, Hickey fired the one marketer they hired for Datomic. He lucked out when a unicorn went all in on it. Word around Cognitect was, Datomic was barely breaking even.
    - xmlblog 3y ago
      > "small-ish backoffice systems that never has to be web scale". Doesn't production use of Datomic by Nubank and Netflix (to mention just two examples) belie this assertion?
      - bilkow 3y ago
btown 3y ago
https://www.postgresql.org/docs/11/contrib-spi.html#id-1.11.7.45.6 https://www.postgresql.org/docs/11/contrib-spi.html#id-1.11.... discusses a model for implementing time travel in Postgres <12 using SPI. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2d10defa7771931295ee804cf096fb9388472a7b https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... discusses why it was removed in Postgres 12 - it seems logical that it's more maintainable to implement in plpgsql, though as far as I can tell there aren't off-the-shelf implementations of this. We use https://django-simple-history.readthedocs.io/en/latest/ https://django-simple-history.readthedocs.io/en/latest/ (with some custom tooling for diff generation) for audit logs and resettability, and while you can't move an entire set of tables back in time simultaneously, it's usually sufficient for understanding data history.

ithrow 3y ago

I guess this is why datomic.com front page now defaults to datomic pro and not cloud.

avodonosov 3y ago

Why backups were impossible? Couldn't you backed up the storage resources?

bvanderveen 3y ago
As far as I could tell, there was no straightforward way to point a new instance of the compute resources back at the old storage resources since they're all provisioned in one CF template.

outworlder 3y ago

Are they _forcing_ you to use CloudFormation? Or is it just the officially supported mechanism? > Mother Postgres can do no wrong. I'll say that Postgres is usually the answer for the vast majority of use-cases. Even when you think you need something else to do something different, it's probably still a good enough solution. I've seen teams pitching other system just because they wanted to push a bunch of JSON. Guess what, PG can handle that fine and even run SQL queries against that. PG can access other database systems with its foreign data wrappers(https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers). The main difficulty is that horizontally scaling it is not trivial(although not impossible, and that can be improved with third party companies).

JohnBooty 3y ago
Yes. Postgres such a reliable and known quantity that IMO it should be the default choice for just about anything. Don't misunderstand me. There are plenty of times when something else is the right choice. I'm just saying, when I have a say in the matter, folks need to clear that bar -- "tell me why tool xyz is going to be so much better than postgres for this use case that it justifies the overhead of adding another piece of software infrastructure." Like, you want to add a document database? Obviously Mongo, Elasticsearch, etc are "best of breed." But Postgres is pretty capable and this team is already good at it. Are we ever going to have so many documents that e.g. Elasticsearch's mostly-effortless horizontal scaling even comes into play? If you don't ever see yourself scaling past 1,000 documents then adding a new piece of infra is a total joke. I see that kind of thing all the time. I can't tell if developers truly do not understand scale, or if they simply do not give a f--- and simply want to play with shiny new toys and enrich their resumes. I mean, I've literally had devops guys telling me we need a Redis cluster even though we were only storing a few kilobytes of data, that was read dozens of times daily with zero plans to scale. That could have been a f'in Postgres table. Devops guy defended that choice hard even when pressed by mgmt to reduce AWS spend. WTF?
- jwr 3y ago
  > Postgres such a reliable and known quantity that IMO it should be the default choice for just about anything. This is being repeated so often. And yet — the above is true, IF (and that's a big if for some of us) you are OK with having your database on a single machine. If you want a distributed database with strict serializability, where some nodes can go down and you still get correct answers, Postgres is not it.

lgrapenthin 3y ago

As someone who is using Datomic Pro in production for many years now I must agree with you. One time I began a project with Datomic Cloud and it was a disaster similar to what you described. I learned a lot about AWS, but after about half a year we switched to Datomic Pro. There were some cool ideas in Datomic Cloud, like IONs and its integrated deployment CLI. But the dev workflow with Datomic Pro in the REPL, potentially connected to your live or staging database is much more interactive and fun than waiting for CodeDeploy. I guess there is a reason Datomic Pro is the featured product on datomic.com again. It appears that Cognitect took a big bet with Datomic Cloud and it didn't take off. Soon after the NuBank acquisition happened. That being said, Datomic Cloud was not a bad idea, it just turned out that Datomic Pro/onPrem is much easier to use. Also of all their APIs, the "Peer API" of Pro is just the best IME, especially with `d/entity` vs. "pull" etc.

ggleason 3y ago

> Datomic Cloud is slow, expensive, resource intensive, designed in the baroque style of massively over-complicated CloudFormation astronautics. Hard to diagnose performance issues. Impossible to backup. You should give TerminusDB a go (https://terminusdb.com/ https://terminusdb.com/), it's really OSS, the cloud version is cheap, fast, there are not tons of baroque settings, and it's easy to backup using clone. TermiusDB is a graph database with a git-like model with push/pull/clone semantics as well as a datalog.

froggertoaster 3y ago

> Mother Postgres can do no wrong. Simple, eloquent, damn true.