4 ms·

I’d be curious to hear how Datomic scales, ops wise and application wise. Is adding a “peer” all there is to do to handle more reads?

by iampims 13y ago

coolsunglasses 13y ago
The more specific you are about your concerns, the better. >Is adding a “peer” all there is to do to handle more reads? Yes, modulo storage backend throughput capacity. It uses storage backends in a fairly dumb and efficient manner so in all likelihood peers will be the bottleneck. >ops wise PostgreSQL and DynamoDB are pretty popular, the ops scenario for the latter is pretty obvious. You can just scale the provisioned IOPS according to your storage backend throughput. Pretty sure DynamoDB can exceed anything you'll ever need in that regard. A sufficiently beefy PostgreSQL machine (SSD, etc) can go ham too. The main shift in mentality with Datomic ops and application-wise is that you're prioritizing vertical scaling before horizontal scaling. I actually think that's the smarter thing to do from a cost and complexity stand-point anyway. I've worked in environments where horizontal scaling was deployed prematurely and we were managing 100 machines where ~3-5 would've sufficed. Part of this enhanced emphasis on scaling is that you're incentivized to make the most of each peer. If you have peers that are making "lightweight" use of (Datomic/your data) but you need many of them running (for compute?) then call into a REST peer or custom REST interface to read/write the data. You implement sprocs through database functions which are just Clojure code, super nice. Main scaling choke-point is just the transactor itself. >Application wise The two X-factors for Datomic in my experience are datalog and the deeply "historical" nature of the database and querying. Being able to arbitrarily time-slice queries with complicated relationships without having to change them in order to accommodate the "time-slicing" is amazing. There are separate interfaces for querying and transacting, which is nice. I would advise avoiding doing more migrations than necessary since that means you'll have to use my library Brambling though :) The Datomic guys are working on implementing trivial schema changes (like adding indexes), but for the more complicated migrations, you'll have to use Brambling or something comparable. Any other queries?
- iampims 13y ago
  Thanks for the detailed reply. One more question: What's your failure recovery strategy when your writer fails?
  - brandonbloom 13y ago
    Just like your traditional SQL deployment with a single master, Datomic supports hot failover for high availability: http://docs.datomic.com/ha.html http://docs.datomic.com/ha.html
- cgag 13y ago
  In a previous thread about datomic, you mentioned trying to have at least 4 gigs of ram, is that ideal for a large app under a lot of load, or basically necessary? Am I going to run into any trouble using it on side projects on my 1 gig VPS? I'd imagine not but that post made me think twice.
  - coolsunglasses 13y ago
    As long as you're not doing anything heavy duty or sensitive, nor addressing large datasets you can squeak by on 1gb, but you might spend a little time tweaking the JVM settings for your app and datomic. Easiest way to squeak by is to not actually start a real transactor but to use an in-memory instance, but that's obviously ephemeral and not going to guarantee your data doesn't go poof.
- dustingetz 13y ago
  Last week in the official cognitect training they said that the transactor can handle 10k writes/sec with appropriate hardware, which is enough to handle all the credit card transactions in the world. [This next part is me, not them] So despite being theoretically write bottlenecked, it still does better than postgres, because the transactor does less work (though they're in the same ballpark). What do you think? I hope I'm quoting them right, from memory.
  - 13y ago
bdunbar 13y ago
I'm not sure how well it scales - the developers seem to be thrilled b it. But this week I'm backing up, and migrating datomic from AWS to our own hosted hardware. Doing this for our dev environment? Man, it Just Worked. I couldn't be happier. I'm still rsyncing the files from AWS to hosted, but I couldn't be a happier systems administrator at this point.

I’d be curious to hear how Datomic scales, ops wise and application wise. Is adding a “peer” all there is to do to handle more reads?

by iampims 13y ago

I’d be curious to hear how Datomic scales, ops wise and application wise. Is adding a “peer” all there is to do to handle more reads?

coolsunglasses 13y ago

The more specific you are about your concerns, the better. >Is adding a “peer” all there is to do to handle more reads? Yes, modulo storage backend throughput capacity. It uses storage backends in a fairly dumb and efficient manner so in all likelihood peers will be the bottleneck. >ops wise PostgreSQL and DynamoDB are pretty popular, the ops scenario for the latter is pretty obvious. You can just scale the provisioned IOPS according to your storage backend throughput. Pretty sure DynamoDB can exceed anything you'll ever need in that regard. A sufficiently beefy PostgreSQL machine (SSD, etc) can go ham too. The main shift in mentality with Datomic ops and application-wise is that you're prioritizing vertical scaling before horizontal scaling. I actually think that's the smarter thing to do from a cost and complexity stand-point anyway. I've worked in environments where horizontal scaling was deployed prematurely and we were managing 100 machines where ~3-5 would've sufficed. Part of this enhanced emphasis on scaling is that you're incentivized to make the most of each peer. If you have peers that are making "lightweight" use of (Datomic/your data) but you need many of them running (for compute?) then call into a REST peer or custom REST interface to read/write the data. You implement sprocs through database functions which are just Clojure code, super nice. Main scaling choke-point is just the transactor itself. >Application wise The two X-factors for Datomic in my experience are datalog and the deeply "historical" nature of the database and querying. Being able to arbitrarily time-slice queries with complicated relationships without having to change them in order to accommodate the "time-slicing" is amazing. There are separate interfaces for querying and transacting, which is nice. I would advise avoiding doing more migrations than necessary since that means you'll have to use my library Brambling though :) The Datomic guys are working on implementing trivial schema changes (like adding indexes), but for the more complicated migrations, you'll have to use Brambling or something comparable. Any other queries?

iampims 13y ago
Thanks for the detailed reply. One more question: What's your failure recovery strategy when your writer fails?
- brandonbloom 13y ago
  Just like your traditional SQL deployment with a single master, Datomic supports hot failover for high availability: http://docs.datomic.com/ha.html http://docs.datomic.com/ha.html
cgag 13y ago
In a previous thread about datomic, you mentioned trying to have at least 4 gigs of ram, is that ideal for a large app under a lot of load, or basically necessary? Am I going to run into any trouble using it on side projects on my 1 gig VPS? I'd imagine not but that post made me think twice.
- coolsunglasses 13y ago
  As long as you're not doing anything heavy duty or sensitive, nor addressing large datasets you can squeak by on 1gb, but you might spend a little time tweaking the JVM settings for your app and datomic. Easiest way to squeak by is to not actually start a real transactor but to use an in-memory instance, but that's obviously ephemeral and not going to guarantee your data doesn't go poof.
dustingetz 13y ago
Last week in the official cognitect training they said that the transactor can handle 10k writes/sec with appropriate hardware, which is enough to handle all the credit card transactions in the world. [This next part is me, not them] So despite being theoretically write bottlenecked, it still does better than postgres, because the transactor does less work (though they're in the same ballpark). What do you think? I hope I'm quoting them right, from memory.
- 13y ago

bdunbar 13y ago

I'm not sure how well it scales - the developers seem to be thrilled b it. But this week I'm backing up, and migrating datomic from AWS to our own hosted hardware. Doing this for our dev environment? Man, it Just Worked. I couldn't be happier. I'm still rsyncing the files from AWS to hosted, but I couldn't be a happier systems administrator at this point.